2

My stellar node stop syncing with such error:

2020-01-15T05:43:46.809 GAES7 [History DEBUG] Verifying ledger headers from /mnt/volume_lon1_01/buckets/tmp/catchup-425e2f74558fb272/ledger/01/a7/8a/ledger-01a78abf.xdr for checkpoint 27757247
2020-01-15T05:43:46.809 GAES7 [History ERROR] Bad ledger-header history entry: claimed ledger [seq=27757241, hash=dd408c] does not agree with LCL [seq=27757241, hash=9ac635] [VerifyLedgerChainWork.cpp:160]
2020-01-15T05:43:46.809 GAES7 [History ERROR] Catchup material failed verification - hash mismatch, propagating failure [VerifyLedgerChainWork.cpp:337]
2020-01-15T05:43:46.809 GAES7 [History ERROR] One or more of history archives may be corrupted. Update HISTORY configuration entry to only contain valid ones [VerifyLedgerChainWork.cpp:339]
2020-01-15T05:43:46.809 GAES7 [History INFO] Verifying ledgers 27757242..27760466
2020-01-15T05:43:46.809 GAES7 [History WARNING] Catchup failed
2020-01-15T05:43:46.809 GAES7 [Ledger ERROR] Catchup will restart at next close. [LedgerManagerImpl.cpp:691]

Restart node does not help. The only thing I can do is to init new db and run node again. But this happens every 2-3 days. Can somebody help me???

Improve this question

2 Answers 2

Reset to default
1

Typically this is the result of some sort of data corruption on the node's local storage system. Which version of stellar-core are you running? Why did it (re)-enter catchup after 2-3 days? Did the node crash by any chance, or just lose sync?

Improve this answer
9
  • Thank you for the answer. I am using the latest version of stellar-core stellar-core 12.2.0 (3f8a5356f3948eff57d17d76f6341ece4d47e2ff) on Ubuntu 18.04. Can't imagine why does it happen. Node never crashes. But this problems started occuring from 31.12.2019. You said that "Typically this is the result of some sort of data corruption on the node's local storage system". But the log is surprising me One or more of history archives may be corrupted. Update HISTORY configuration entry to only contain valid ones [VerifyLedgerChainWork.cpp:339] And I can not find what configs should I update.
    – alexqrid
    Commented Jan 16, 2020 at 11:53
  • As I got the history archives from SDF nodes that log message made me think that something happened with SDF nodes. So I added additional nodes to fetch history archives from, but after a couple of days I met this problem again.
    – alexqrid
    Commented Jan 16, 2020 at 12:01
  • Interesting! this presents 3 possibilities (listed in what I think are the order of likelihood): 1. Your node's local disk or network corrupted some bits in transit. Possibly a faulty disk, network controller or such. Can you run disk diagnostics? For example on ubuntu, apt install smartmontools and run smartctl -a /dev/<the-disk> and look for the SMART overall-health self-assessment test result? 2. The history archives you're fetching from have corrupt data. 3. Stellar-core has a bug in its data validation routines and is failing to validate actually-valid data.
    – Graydon Hoare
    Commented Jan 17, 2020 at 17:56
  • (I'll check on the archives and verify the hash you're seeing)
    – Graydon Hoare
    Commented Jan 17, 2020 at 17:57
  • I've checked multiple archives and they all appear to have dd408c8785919ab2a6f774c1b81763f6bcf3f32446ffcc6bdce4bb28fed85adf as the hash for ledger 27757241 -- so given multiple other nodes are regularly doing full network-validated catchups from those archives, the data is probably correct in the archives and you're dealing with either corruption on your end or a bug in core.
    – Graydon Hoare
    Commented Jan 17, 2020 at 18:29
0

The problem solved. It seems that there was a problem with the network. For more information refer to the issue on github

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.