Testing Stellar for high volume of operations per second

Question

I'm testing Stellar for a project where we want to be able to get ~4000 operations per second. Multiple sources ( including this question's answer: Scalability on Stellar network ) indicate that I should be able to achieve this on a private Stellar network, yet when I try to run tests with 100 ops/tx and 200 txs/ledger, I end up with lots of errors and failed transactions instead of the ahem stellar results I was hoping for.

Has anyone actually gotten this many ops/s to work? Do you have any tips?

---

More details:

I'm working on Linux, and the load testing program I'm using came from here: https://github.com/kinecosystem/stellar-load-testing

Using that program, I can get 100ops/tx to work if my number of accounts is low, but I end up with very low txs/s. I can get 40 txs/s, but only with a low number of ops/tx.

The errors I'm getting most often are:

1. "http post failed: Post http://localhost:8000/transactions: net/http: request canceled (Client.Timeout exceeded while awaiting headers)" -- I assume this is probably a timeout somewhere in my system, rather than a problem with Stellar, but I'm still trying to track down where it's coming from.

2. "Horizon error: \"Internal Server Error\". Check horizon.Error.Problem for more information.", followed by "failed to extract result codes from horizon response"

And just for the record: I'm still a bit green as an engineer, so don't hesitate to suggest if you think I missed something really basic!

---

UPDATE: I did end up finding a timeout that was set within the load test I was running, and got some better results. Will update more when I've done some further testing.

Answer 1

Start the performance troubleshooting from pinpointing a bottleneck. During the test execution run top to get the list of the top resource consuming processes. There are 4 possible options:

1. Your load testing app consumes 100% of CPU. Run the test loader app on a separate machine connected to the same local network to test Steall performance instead of test loader performance and eliminate slow network impact.
2. PostgreSQL consumes the most of CPU. Tune up your database settings and memory limits.
3. Horizon is a top resource consumer. Try to send transactions directly to the Stellar Core instance avoiding Horizon, as it actually works as a mediator in your scenario.
4. Stellar Core consumes 100% of CPU. It looks like you reached the maximum possible tps rate for your hardware.

Also check the memory footprint and disks IO utilization. If more than 70% of all installed memory is taken, OS may use swap files, which is one of the main performance killers. Needles to say that running such tests on the machine with HDD drives instead of SSD makes no sense at all. So if you are running tests on your laptop, it wouldn't give the same results as dedicated server with at least 32GB memory and SSD drives.