Horizon gives 504 after firing many transactions

Question

I have three stellar-core instances and one Horizon server. The Horizon server is connected to the first of three nodes.

All goes well for a while, until I start firing many, many transactions at Horizon (Stellar advertises being able to process 1k transactions/s, so I'm firing 1000 individual transactions at Horizon.)

At this point, I start receiving 400-bad request errors from Horizon, and looking at the Horizon logs shows many 504 errors.

When I do a "health check" on Stellar, all three nodes are synched, and no nodes say that other nodes are missing. In other words, everything looks dandy from the Stellar point of view.

Do I need to run multiple Horizon instances to be able to fire 1k transactions at the thing? I'm running the three nodes and Horizon on a machine with 20 cores and SSD. Stellar and Postgresql are all writing to the SSD.

Answer 1

Stellar is able to process around 1000 operations per seconds, or around 10 transactions per second (with current version of Stellar Network protocol, allowing maximum 50 tx per ledger, and average ledger time of about 5 seconds).

Account sequence is another restriction, it is incremented each time the transaction is executed.

Each transaction has a sequence number. Transactions follow a strict ordering rule when it comes to processing of transactions per account. For the transaction to be valid, the sequence number must be 1 greater than the sequence number stored in the source account entry when the transaction is applied. After the transaction is applied, the source account’s stored sequence number is incremented by 1. If the sequence number on the account is 4, then the incoming transaction should have a sequence number of 5. After the transaction is applied, the sequence number on the account is bumped to 5.

Note that if several transactions with the same source account make it into the same transaction set, they are ordered and applied according to sequence number. For example, if 3 transactions are submitted and the account is at sequence number 5, the transactions must have sequence numbers 6, 7, and 8.

For each ledger you can submit more than one transaction with the same sequence for a given source_account, but only one of them will be executed, others will fail.

In order to get higher throughput, you might want to include more than one operation into a transaction, and use multiple source accounts (so called "channels" approach).

As for 504 errors, that's a Gateway Timeout Error. Could you submit error details? It may be an error of Horizon or Stellar Core itself.

Answer 2

I met the same issue here. About the sequence number, you should use channel instead or manage the sequence number locally, not the sequence number that stellar core send back to you. An other issue is after a request time-out, the next request for the account will be bad sequence. I seem that issue appear here: tx_bad_seq error when creating account with the root account of Stellar private network.

By the way, could you show me how to set up for a three stellar-core instances and one Horizon server. I manage to try with https://github.com/stellar/docker-stellar-core and build horizon my self. But i can't get the history of stellar core.