0

I have a history of stellar core and horizon databases which we abandoned a year ago. Now we need to catch up with the blockchain. stellar core synched to some height, and Horizon lagging few million ledgers behind. We have updated to new stellar core 16 and horizon 2.1.1. When I learned about captive core, I tried running new Horizon in this mode. It would download zip files with historical info with the help of captive core and load them into postgres. However, it says commit=false for each ledger it loads into the DB, and indeed transaction is not committed. Horizon spent about a week in that state, processed million ledgers, and then downloaded another batch of ZIP files, complained that few files are missing, and stuck. No ledgers are being loaded, transaction is still not committed. Postgres was doing autovacuum in Horizon DB for some time, but then it finished, and the system did not have any load (neither CPU nor I/O) except for captive core which was using 10% of CPU and steadily growing memory (not surprising as it keeps ledgers it got from history in memory). When horizon is requested to provide any information via JSON API (the simplest one, like latest ledger), it responds with HTTP 503.

After two days in that state I gave up, stopped Horizon, changed config to disable captive core (--enable_captive_core_ignestion=false), and started both stellar core and horizon. Still horizon does not commit DB transaction after writing the ledger:

time="2021-05-03T07:22:29.827Z" level=info msg="Processing ledger" commit=false ledger=true pid=19622 sequence=30685453 service=ingest state=false

I can cross-check the fact that DB transaction is not committed by requesting latest known ledger from horizon via JSON RPC (ledgers?limit=1&order=desc) or by querying count of rows in history_ledgers table - both stay the same though horizon progresses through the ledgers. iotop shows 50% of I/O utilization by postgres horizon DB, doing INSERT.

My concern is that it will never commit, or will commit after few million ledgers, and postgres will take forever to commit (we are running in Azure where "Premium SSD" is not SSD and is quite slow). Is there a way to force it to commit DB transaction after each ledger or some batch of ledgers?

Stellar core is not fully synced, it is catching up. Might it be the reason of not committing Horizon's DB transaction?

Saw something suspicious in captive core log when downloading historical archives:

time="2021-05-01T04:23:10.500Z" level=warning msg="Process: process 18602 exited 22: curl -sf http://history.stellar.org/prd/core-live/core_live_001//ledger/02/18/df/ledger-0218dfff.xdr.gz -o buckets/tmp/ca$
time="2021-05-01T04:23:10.500Z" level=error msg="History: Could not download file: archive h0 maybe missing file ledger/02/18/df/ledger-0218dfff.xdr.gz" pid=5753 service=ingest subservice=stellar-core

Notice two slashes: core_live_001//ledger

URL is configured per:

Stellar Core cfg file

HISTORY="curl -sf https://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

Also, tried these instructions: Reingest with captive core

But got:

$ stellar-horizon db reingest range --parallel-workers=24 17000000 17001000
2021/05/03 12:49:43 successfully applied 47 horizon migrations
2021/05/03 12:49:43 Invalid config: one or more captive core params passed (--stellar-core-binary-path or --captive-core-config-append-path) but --ingest not set. If you are migrating from Horizon 1.x.y read the Migration Guide here: https://github.com/stellar/go/blob/master/services/horizon/internal/docs/captive_core.md

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.