We are trying to setup a captive core with Horizon API server with historical data as far as block 28960000 (since our token got used since then). Installation and initial sync went smooth but ingesting historical data takes a long time. We run this on the following hardware:

  • Dual CPU E5-2660 v4 = 28 cores / 56 threads
  • 192GB RAM
  • Root on a md software raid 1 with 2 x 1TB SSD
  • PostgreSQL database on a btrfs software raid 5 with 6 x 2 TB Samsung 870 QVO SSD's

Everything was setup from scratch: Ubuntu, postgres and the software raid's. We started the ingest with the following commands, which are still running fine currently. The issue is these have been running for 14 day's already now:

  • stellar-horizon db reingest range --parallel-workers=20 36000000 37000000
  • stellar-horizon db reingest range --parallel-workers=20 35000000 36000000

Before starting the ingest we set the following variables:

  • export DATABASE_URL=postgres://user:password@localhost:5432/horizon
  • export APPLY_MIGRATIONS=true
  • export HISTORY_ARCHIVE_URLS=https://s3-eu-west-1.amazonaws.com/history.stellar.org/prd/core-live/core_live_001
  • export NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"
  • export STELLAR_CORE_BINARY_PATH=$(which stellar-core)
  • export PARALLEL_JOB_SIZE=100000
  • export RETRIES=10

Some server stats: server stats

We are not able to find issues or errors. This is the current ingest: ingest jobs

Everything seems to be running fine, only that the above two ingest jobs have already been running for 14 day's now. As we understand, ingesting on a system like this (with much more then 15k IOPS) should maybe take a few days to ingest all historical data. We know that write IOPS do not stack in a raid 5, but one of those samsung SSD's have a 8K write IOPS. So we are not sure where the bottleneck is. Maybe we oversee something? Please advice.

Any help would be greatly appreciated! As we planned to have this server up in a few days. More questions / need more data? Ask and we will provide! Thanks in advance!

  • We got a lot of performance improvements after switching from btrfs to xfs and from a soft RAID5 to a RAID0. Still ingesting one of the most recent million blocks takes between 14 and 16h. We where also ingesting with 20 parallel workers, we found that using 15 workers also speeds up ingest by a few hours. Limitations do not seem to be hardware / setup now, since we are not nearly using all available iops in the setup while ingesting. Could someone elaborate how it is even possible to ingest all the ledgers in 1 to 3 day's? Sep 27, 2021 at 12:28


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.