I have three stellar-core instances and one Horizon server. The Horizon server is connected to the first of three nodes.
All goes well for a while, until I start firing many, many transactions at Horizon (Stellar advertises being able to process 1k transactions/s, so I'm firing 1000 individual transactions at Horizon.)
At this point, I start receiving 400-bad request errors from Horizon, and looking at the Horizon logs shows many 504 errors.
When I do a "health check" on Stellar, all three nodes are synched, and no nodes say that other nodes are missing. In other words, everything looks dandy from the Stellar point of view.
Do I need to run multiple Horizon instances to be able to fire 1k transactions at the thing? I'm running the three nodes and Horizon on a machine with 20 cores and SSD. Stellar and Postgresql are all writing to the SSD.