Performance problem with ingesting Horizon

Question

We want to setup Stellar Full Archive with Horizon

Using the https://github.com/satoshipay/stellar-core-parallel-catchup we managed to synchronized Stellar Core DB and build full archive.

After lunching new instance, from the gathered data we lunched Stellar Horizon. But Horizon was only able to serve only new leadgers (those, which one wasn't been in snapshot).

So after some googling we run stellar-horizon db reingest start end, in parallel way - 4 concurrent threads, chunk size 10k, but the performance is really bad. After one day of work, we estimate that the whole process will take about 100 days. Which is not acceptable for us.

Could anyone kindly advice which parameters should by tunned (for horizon, postgresql or even OS)? CPU is utillized in about 30%, %iowait 6% to 24%, Memory usage 3GB of 16GB.

Is it possible, to reingest horizon on separate environment in parallel way and merge data to the production environment (something simillar like for Stellar Core DB and History Archive)? Which tables need to be coopied?

Answer 1

We have an experimental feature that we're going to beta-release this month. It's called Captive Stellar-Core and you can use it now (requires passing a special feature flag) to reingest full history much faster compared to how Horizon operates now (full reingestion in less than two days).

It requires a larger instance (in terms of CPU and memory) but only for reingestion. Details can be found in this doc.