2

Settings: Private network, 4 nodes running good and validating each other. Now I try to add a 5th node for testing, upon encountering an Horizon ingestion issue.


this is how I made the node "5a" (there is also a "5b")

Step 1, made a copy from the one of the 4 existing nodes

Step 2, clear all DB ; IP changed ; NODE_SEED changed ; stellar-core.cfg set to have 5 validators (4 existing + itself) and CATCHUP_COMPLETE=true

Step 3, reboot and start this 5th node

After a while, the following appeared at the Horizon log:

Looks strange that

time="2018-03-15T11:07:29+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1593
time="2018-03-15T11:07:30+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1593
time="2018-03-15T11:07:31+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1593
time="2018-03-15T11:07:32+08:00" level=info msg="history db is empty, establishing base at ledger 2" pid=1593
time="2018-03-15T11:07:36+08:00" level=info msg="ingest: already in progress" pid=1593
time="2018-03-15T11:07:38+08:00" level=info msg="ingest: already in progress" pid=1593
time="2018-03-15T11:07:40+08:00" level=info msg="ingest: already in progress" pid=1593

As it looked strange that Horizon counted from ledger 2, I repeated the above steps except the IP was different. At this node "5b" (also only 5 validators), Horizon started its base from ledger 41.

time="2018-03-15T12:01:38+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1592
time="2018-03-15T12:01:39+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1592
time="2018-03-15T12:01:40+08:00" level=warning msg="ingest: waiting for stellar-core sync" pid=1592
time="2018-03-15T12:01:41+08:00" level=info msg="history db is empty, establishing base at ledger 41" pid=1592
time="2018-03-15T12:01:45+08:00" level=info msg="ingest: already in progress" pid=1592
time="2018-03-15T12:01:47+08:00" level=info msg="ingest: already in progress" pid=1592
time="2018-03-15T12:01:49+08:00" level=info msg="ingest: already in progress" pid=1592

Any idea what went wrong..?

  • Just set up one more control experiment: level=info msg="history db is empty, establishing base at ledger 86". These varying numbers are perplexing... – cesarm Mar 16 '18 at 1:55
3

Given how stellar-core populates its txhistory and txfeehistory tables (as a queue containing arbitrary gaps), the horizon history system establishes its elder ledger at the stellar-core latest ledger when initializing an empty horizon database. This behavior is to help reliability and lower admin intervention across most configurations. To backfill the skipped ledgers, run the horizon db backfill COUNT command against your installation.

While not specifically mentioning this behavior, the reference docs for ingestion are here: https://www.stellar.org/developers/horizon/reference/admin.html#ingesting-stellar-core-data

  • Thanks Scott, I will try your solution later when I came back to this issue. Last week I just reset everything as I observed this behavior. – cesarm Mar 26 '18 at 1:29
  • Is it possible to not skip any ledgers? Backfilling takes days/weeks. – relik Oct 29 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.