We are trying to setup a captive core with Horizon API server with historical data as far as block 28960000 (since our token got used since then). Installation and initial sync went smooth but ingesting historical data takes a long time. We run this on the following hardware:
- Dual CPU E5-2660 v4 = 28 cores / 56 threads
- 192GB RAM
- Root on a md software raid 1 with 2 x 1TB SSD
- PostgreSQL database on a btrfs software raid 5 with 6 x 2 TB Samsung 870 QVO SSD's
Everything was setup from scratch: Ubuntu, postgres and the software raid's. We started the ingest with the following commands, which are still running fine currently. The issue is these have been running for 14 day's already now:
- stellar-horizon db reingest range --parallel-workers=20 36000000 37000000
- stellar-horizon db reingest range --parallel-workers=20 35000000 36000000
Before starting the ingest we set the following variables:
- export DATABASE_URL=postgres://user:password@localhost:5432/horizon
- export APPLY_MIGRATIONS=true
- export HISTORY_ARCHIVE_URLS=https://s3-eu-west-1.amazonaws.com/history.stellar.org/prd/core-live/core_live_001
- export NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"
- export STELLAR_CORE_BINARY_PATH=$(which stellar-core)
- export ENABLE_CAPTIVE_CORE_INGESTION=true
- export PARALLEL_JOB_SIZE=100000
- export RETRIES=10
- export RETRY_BACKOFF_SECONDS=20
Some server stats:
We are not able to find issues or errors. This is the current ingest:
Everything seems to be running fine, only that the above two ingest jobs have already been running for 14 day's now. As we understand, ingesting on a system like this (with much more then 15k IOPS) should maybe take a few days to ingest all historical data. We know that write IOPS do not stack in a raid 5, but one of those samsung SSD's have a 8K write IOPS. So we are not sure where the bottleneck is. Maybe we oversee something? Please advice.
Any help would be greatly appreciated! As we planned to have this server up in a few days. More questions / need more data? Ask and we will provide! Thanks in advance!
