We run a remote Postgres database on our Horizon instance, and on several occasions we would lose connection to the database server briefly (cause still being investigated). This drops our connection to the database itself, and from that point on Horizon is non-functional. It still processes transactions, but any queries fail with a 500 error. When looking in Horizon log, I see repeated errors:
May 26 13:47:14 NODE1 stellar-horizon[24626]: time="2021-05-26T13:47:14.892Z" level=error msg="Error getting state invalid value" err="could not get value: get failed: write tcp xxx.xx.xxx.xx:42536->xx.xx.xxx.xxx:5432: write: connection reset by peer" pid=24626 service=ingest
Restarting Horizon fixes the issue, but I wonder - is there a built-in recovery/retry logic for database connection? If so, how would I enable it? And if not, is it to feasible implement something within Horizon that is more resilient?
At this point, until we solve network glitch, the only option we have is to detect the error in the log and trigger recycle of Horizon service. Are there any other options available to us?
