1

Document https://github.com/stellar/go/blob/master/services/horizon/internal/docs/admin.md says:

To reingest older ledgers (due to a version upgrade) or to ingest ledgers closed by the network before you started Horizon. This is done through the horizon db range [START_LEDGER] [END_LEDGER] command, which could be run as follows:

horizon1> horizon db reingest range 1 10000
horizon2> horizon db reingest range 10001 20000
horizon3> horizon db reingest range 20001 30000

This allows reingestion to be split up and done in parallel by multiple Horizon processes.

Running ingestion in parallel was my intention, but for me, Horizon v1.0.0 (which recently became available) does not start loading second range until the first range is loaded. For example, now I have a process which ingests ledgers in range 20000001-21000000. In another session, I run:

./horizon db reingest range 25000001 26000000

It outputs:

INFO[2020-03-04T14:19:03.133Z] Ingestion system initial state                current_state="reingestHistoryRange(fromLedger=25000001, toLedger=26000000, force=false)" pid=22948 service=expingest
INFO[2020-03-04T14:19:03.148Z] Processing ledger                             commit=false graph=false ledger=true pid=22948 sequence=25000001 service=expingest state=false

It sits in this state until the first ingestion is completed (which is for awhile), and only then starts ingesting the ledgers. If it matters, I have the following in stellar-core's config file:

KNOWN_CURSORS=["HORIZON"]

Is there a way of ingesting history in parallel with Horizon v1.0.0?

I have tried running two processes in parallel, and got the following deadlock in PostgreSQL:

2020-03-04 23:53:12.012 UTC [21575] horizon@horizon ERROR:  deadlock detected
2020-03-04 23:53:12.012 UTC [21575] horizon@horizon DETAIL:  Process 21575 waits for ShareLock on transaction 49443; blocked by process 22965.
        Process 22965 waits for ShareLock on transaction 43142; blocked by process 21575.
        Process 21575: INSERT INTO history_accounts (address) VALUES ($1),($2),($3),($4),($5),($6) ON CONFLICT (address) DO NOTHING
        Process 22965: INSERT INTO history_accounts (address) VALUES ($1),($2),($3),($4),($5),($6),($7),($8),($9),($10),($11),($12),($13) ON CONFLICT (address) DO NOTHING
2020-03-04 23:53:12.012 UTC [21575] horizon@horizon HINT:  See server log for query details.
2020-03-04 23:53:12.012 UTC [21575] horizon@horizon CONTEXT:  while inserting index tuple (5596,64) in relation "history_accounts"
2020-03-04 23:53:12.012 UTC [21575] horizon@horizon STATEMENT:  INSERT INTO history_accounts (address) VALUES ($1),($2),($3),($4),($5),($6) ON CONFLICT (address) DO NOTHING
  • 1
    I checked it locally and it works for me. Can you check what's the maximum number of open connections in your Postgres DB? If it's too low and you make many connections, the clients will have to wait. – Bartek Nowotarski Mar 5 at 10:15
  • Thank you! Are you using Horizon 1.0.0? For me, waiting session eventually started loading ledgers, but then the first session of Horizon failed with deadlock in PostgreSQL. I made a conclusion that they won't work in parallel. I use PostgreSQL 12 and my HW is capable enough, number of connection is limited quite high. – DimaA6_ABC Mar 6 at 11:13
  • Yes, I checked with Horizon 1.0.0. Can you post a minimal set of two ledger ranges I could check on my machine? Can you copy the deadlock error you got here? – Bartek Nowotarski Mar 6 at 15:23
  • First was 20000001-21000000, second 25000001-25100000. Deadlock is too long, will add to my question. – DimaA6_ABC Mar 7 at 17:57
  • 1
    OK, this is surprising because we used this before. But it's definitely a bug. Tracking in: github.com/stellar/go/issues/2370 – Bartek Nowotarski Mar 9 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.