Ive successfully installed and started horizon 2.0 but I am having problems reingesting ledgers. Id like to reingest ledgers from [30000000, present].

Ive attempted to do so following the instruction from the Stellar team here:

Stellar Reingestion instructions

The process runs for almost 1 day and eventually fails with one of the following errors:

"error when processing [30227165, 30294876] range: error preparing range: Error fast-forwarding to 30227165: stellar core exited unexpectedly: signal: killed"

" Error creating ledger reader: error getting ledger from the backend: stellar core exited unexpectedly: signal: killed"

The command im using to run the reingest (ive tried different worker counts):

stellar-horizon db reingest range --parallel-workers=5 --network-passphrase="Public Global Stellar Network ; September 2015" --db-url="postgres://user:password@localhost:5432/horizon" --enable-captive-core-ingestion --apply-migrations --captive-core-config-append-path="/etc/default/stellar-captive-core-stub.toml" --history-archive-urls="https://s3-eu-west-1.amazonaws.com/history.stellar.org/prd/core-live/core_live_001" --stellar-core-binary-path="/usr/bin/stellar-core" 30000000 34581449

Oddly enough when the job finishes, the output says it ingestion only a small subset. I expected given the 24 hours that it would have been much larger.

INFO[2021-03-24T01:54:48.734Z] Reingestion done duration=83853.399425002 from=33477341 pid=2672 service=ingest to=33545052

INFO[2021-03-24T01:54:48.734Z] Shut down pid=2672 service=ingest

INFO[2021-03-24T01:54:48.734Z] successfully reingested range from=33477341 pid=2672 service=ingest to=33545052

The instructions on the repo say its possible the ingestion fails and a recommended range would be provided to continue. I think something else is happening here because the range provided is the original range, and again ive tried this 3-4 times now.

Appreciate any help here from the Stellar team and community.

  • Are you running these workers on the same machine? What is the machine spec? Mar 26, 2021 at 20:08
  • @ire_and_curses yes same machine. im running a digital ocean droplet. cpu-optimized (dedicated) / 16 GB / 8 vCPUs. The db has a 1TB volume attached. There were some spikes in cpu usage when the worker count was greater but after i reduced it the load stabilized. Memory was high most of the time.
    – jiujiteiro
    Mar 26, 2021 at 20:22
  • 1
    Yeah it sounds like it should be enough. Each worker could use 3GB though so it's close. What happens if you run with less workers (e.g. 4)? You can test on a smaller range. Mar 26, 2021 at 21:55


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.