Stellar-core was running smoothly, then crashed without log entries. Node now fails to catch-up and sync. Logs are complaining about a mismatched ledger hash and I am stuck in an endless loop of failed replays. It looks like ledger 22013310 might be corrupted?
Is there any known solution to this type of situation? This happened on Stellar-Core v10.0.0.
Update: I saw in the release notes to v10.1.0 that "In some rare circumstances, some buckets would be deleted on startup, causing the instance to be in a corrupt state."
Updated to 10.1.0 and hope this will help.
2019-01-17T17:11:17.765 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] produced mismatched ledger hash [seq=22013310, hash=f9cd73] [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:11:18.767 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:11:21.769 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:11:28.770 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:11:44.771 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:12:07.773 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:16:44.529 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:16:45.530 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:16:49.536 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
2019-01-17T17:16:50.537 GD6SZ [History ERROR] Replay failed: replay of [seq=22013310, hash=9cc7b6] at LCL [seq=22013310, hash=f9cd73] disagreed on hash [ApplyLedgerChainWork.cpp:269]
