7

On a private network, I have 5 core nodes and 5 horizon instances, all of the horizon services were not configured with "history-retention-count", so according to this, this variable to should default to 0, which means that the history should not be deleted

"the minimum number of ledgers to maintain within horizon's history tables. 0 signifies an unlimited number of ledgers will be retained"

However, my horizon instances did not save all of the history,
their "history_elder_ledger" is ~400,000 (changes between each horizon instance).

Unlike the public horizon, where the "history_elder_ledger": 1

What can cause this? and how can I fix it so that I will always be able to get the entire history from horizon?

Improve this question

2 Answers 2

Reset to default
7

I think I had run into the same issue, as Stellar Core by default uses a "fast-sync" mode (instead of "full-sync").

In terms of architecture, Horizon just takes the data stored at Core's DB, i.e. data that Core has processed. **It may be your Core did not process the record before ledger#400000, or at least your Core's DB does not possess the relevant entries. **

By default, Core is doing the CATCHUP_RECENT mode (i.e. "fast-sync"), as CATCHUP_COMPLETE=false.

If a Stellar Core did not start working from ledger#1 (i.e. joining after the network is up, for example at ledger#400005), the Core will catch up at the next checkpoint, but not handle the old old ledgers. It may contain very recent records like ledger#400001 but not #500. As Core did not process #500, Horizon will be unable to take in this record.

Similar problem will occur, when your Core was in sync with the network but is down for a period of time. Using CATCHUP_RECENT mode will not let you get back the data during your node's downtime.

Solution:

Configure Core with CATCHUP_COMPLETE=true, and reset your Core (clear DB, and do things like --newdb & --newhist). Core will then process all the ledgers from #1 to #current, even when it has downtime and did not get certain ledgers.

I would recommend to do the steps node-by-node, so that your previous data is not lost. And it may take long long time in case something went wrong.

From my experience, it took 30min for a new-born node to catchup completely with my 1-week-old network.

After 10 minutes, Core will have entered "Catching up" state and started to process data, saying Publishing 166 queued checkpoints [63-10623]. Horizon should be constantly renewing like "history_latest_ledger": 576, "history_elder_ledger": 31, "core_latest_ledger": 1998550.

After some more time, Core should enter "Synced!" state before Horizon fully catches up. Horizon is ok done when "history_elder_ledger" = "core_latest_ledger".

Full-sync takes time, just come back later.

Improve this answer
12
  • p.s. after full-sync, your "history_elder_ledger" may not be "1". But feel no panic, the transaction history should still be complete.
    – cesarm
    Commented May 2, 2018 at 2:54
  • Is it possible that the Core database cleans itself every x amount of time? I'm looking at the SQL database and it really is not complete all the way back to ledger 1
    – Ron Serruya
    Commented May 2, 2018 at 7:25
  • Core database DOES clean itself. By default, it cleans up the histories every 14400s (=4hrs). The histories are the largest data at Core database, and the smaller data is not cleaned up. I tried with doing a constant-rate stress test at my private network, and the storage count showed that the Core DB shrank almost back to its size every four hours. (Ref: github.com/stellar/stellar-core/blob/master/docs/…)
    – cesarm
    Commented May 2, 2018 at 7:52
  • It is actually rather difficult for you to read/extract from the Core DB. Horizon was designed to have higher readability for users, so I would recommend you to check Horizon DB instead. By default, Horizon DB does not automatically cleans up itself, so the data is strictly growing.
    – cesarm
    Commented May 2, 2018 at 7:54
  • 1
    Found a solution! :) horizon db backfill NUM_LEDGERS set num_ledgers slightly higher than current ledger on stellar-core. This will make horizon go backwards in the history and ingest everything since the beginning of time (as far as it's present in the stellar-core history).
    – marcinx
    Commented Nov 14, 2018 at 9:48
1

Had the same problem but it seems I figured it out. If you want the full history available on your horizon you need to do the following.

Run your stellar-core with CATCHUP_COMPLETE=true. On your horizon side you need to manually tell it to ingest historical ledgers. Just starting horizon with --ingest=true is not enough as it will only ingest ledgers that stellar-core currently processes. The steps to manually ingest history are as follows.

  1. horizon db init initialized db schema
  2. horizon db backfill NUM_LEDGERS set this slightly higher than the current ledger on stellar-core. This way it should back fill all the way to the beginning of history.
  3. horizon --ingest=true Start your horizon with ingest=true to keep it up to date. Horizon can run while you are doing the db backfill in the background.

Maybe there is easier/better ways to do it but this is what worked for me. The docs all just mention --ingest=true but you really also need to do the db backfill for full/more complete history.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.