Best way to test maximum possible TPS on private network?

Question

I have setup a private network and want to do max possible TPS(Transactions per second). Hardware is not an issue, working with GCP and can use 160 core CPU and 4000GB RAM for my machine.

A few questions :

1. What is best way to test the max possible TPS?
2. What is stress test and possible output ? I did run this test but it continuously runs for multiple hours and there is no output.
3. What other test do we have for scalability ?
4. Any example or document for the above will help more.

Answer 1

Well, with your setup TPS rate depends on your private network configuration.

First of all, check maximum transactions per ledger parameter. If you have local Horizon node up and running, just navigate to https://horizon.stellar.org/ledgers/?order=desc&limit=1 (use your Horizon address instead of horizon.stellar.org). Locate the max_tx_set_size parameter.

The maximum number of transactions the validators have agreed to process in a given ledger. If more transactions are submitted than this number, the validators will include those with the highest fees.

You can use this value as a factor of parallelism in your loading tests.

I would advise you the following test setup:

1. Create and fund N accounts, where N is equal to max_tx_set_size value.
2. Prepare the test code. Use endless loop for this code (or you might want to limit the test to, say, 100 iterations).
3. Run the performance loading code in N parallel threads, with N accounts. In this way you will fully utilize the throughput of the Network.
4. Gather CPU and Memory metrics.

Repeat steps 2-4 with each test case. Test cases:

1. Each transaction contains only one simple operation (for example, set random home_domain for the account).
2. Each transaction includes 100 operations (that's a hard-coded limit). For example, you can send 0.000001 XLM to another account 100 times for each transactions.
3. The same as previous, but experiment with different operation types. For example, PATH_PAYMENT or MANAGE_OFFER operations potentially could be more resource consuming than PAYMENT operation.

Once you gathered all required metrics for this setup, initialize a soft fork to increase max_tx_set_size parameter (you can try 200, 500, 1000, 10000). Repeat the test with upgraded Network. Do not forget to create more test accounts to follow the N_test_accounts=max_tx_set_size and set the level of parallelism accordingly.

One more thing. Tests may be inaccurate with all test nodes located in one datacenter. Try to deploy the globally distributed configuration. For example, US - Germany - Japan.