4

I see messages like this in the logs of Stellar Core:

Ledger 20144734 (da1765) closed and could NOT be fully validated by validator

What does this mean? Not sure if it matters, but the core node is almost always 20 to 50 ledgers behind. I'm trying to figure out, and maybe this has something to do with it.

version: 10.0.0

  • Do you have Horizon connected to your stellar-core instance? There's a bug in asset stats code in Horizon that can make connected stellar-core instance slow (and this can make it out of sync). Horizon 0.14.1 has a fix for this: github.com/stellar/go/releases/tag/horizon-v0.14.1 (you need to set environment variable to set it off). – Bartek Nowotarski Sep 24 '18 at 12:08
  • Did you find a solution? I have the same problem on v10.0.0 (stellar-horizon not connected yet). Full validator with CATCHUP_COMPLETE=true – marcinx Oct 18 '18 at 23:21
2

As I see it, it's just a warning for full validators:

if (getSCP().isValidator() && !validated)
{
    CLOG(WARNING, "Herder")
        << fmt::format("Ledger {} ({}) closed and could NOT be fully "
                       "validated by validator",
                       slotIndex, hexAbbrev(value.txSetHash));
}

(From https://github.com/stellar/stellar-core/blob/v10.0.0/src/herder/HerderImpl.cpp#L184)

As the node is restarted and could not validate the transaction at the time it was up for validation, it will never be validated.

For example, after stopping a full validator then running it again:

2018-10-31T08:20:50.185 GD5VV [Ledger INFO] Loaded last known ledger: [seq=26, hash=f3c80e]
2018-10-31T08:20:52.350 GD5VV [Herder INFO] Quorum information for 25 : {"agree":3,"disagree":0,"fail_at":2,"hash":"31f2a5","missing":0,"phase":"EXTERNALIZE","validated":false}
2018-10-31T08:20:52.350 GD5VV [Herder WARNING] Ledger 27 (498bc9) closed and could NOT be fully validated by validator
2018-10-31T08:20:52.359 GD5VV [Ledger INFO] Got consensus: [seq=27, prev=f3c80e, tx_count=0, sv: [  txH: 498bc9, ct: 1540970409, upgrades: [ ] ]]
2018-10-31T08:20:52.368 GD5VV [Ledger INFO] Closed ledger: [seq=27, hash=4584eb]
2018-10-31T08:20:52.368 GD5VV [Ledger INFO] Changing state LM_BOOTING_STATE -> LM_SYNCED_STATE
2018-10-31T08:20:52.370 GD5VV [Herder INFO] Quorum information for 26 : {"agree":3,"disagree":0,"fail_at":2,"hash":"31f2a5","missing":0,"phase":"EXTERNALIZE","validated":true}
2018-10-31T08:20:52.376 GD5VV [Ledger INFO] Got consensus: [seq=28, prev=4584eb, tx_count=0, sv: [  txH: 68b830, ct: 1540970414, upgrades: [ ] ]]
2018-10-31T08:20:52.385 GD5VV [Ledger INFO] Closed ledger: [seq=28, hash=834d8b]
2018-10-31T08:20:52.395 GD5VV [Herder INFO] Quorum information for 27 : {"agree":3,"disagree":0,"fail_at":2,"hash":"31f2a5","missing":0,"phase":"EXTERNALIZE","validated":false}
2018-10-31T08:20:52.401 GD5VV [Ledger INFO] Got consensus: [seq=29, prev=834d8b, tx_count=0, sv: [  txH: d24f79, ct: 1540970419, upgrades: [ ] ]]
2018-10-31T08:20:52.410 GD5VV [Ledger INFO] Closed ledger: [seq=29, hash=3ff594]
2018-10-31T08:20:52.420 GD5VV [Herder INFO] Quorum information for 28 : {"agree":3,"disagree":0,"fail_at":2,"hash":"31f2a5","missing":0,"phase":"EXTERNALIZE","validated":true}
2018-10-31T08:20:52.476 GD5VV [Ledger INFO] Got consensus: [seq=30, prev=3ff594, tx_count=0, sv: [  txH: 676009, ct: 1540970424, upgrades: [ ] ]]
2018-10-31T08:20:52.749 GD5VV [Ledger INFO] Closed ledger: [seq=30, hash=b2d05a]
2018-10-31T08:20:52.795 GD5VV [Herder INFO] Quorum information for 29 : {"agree":3,"disagree":0,"fail_at":2,"hash":"31f2a5","missing":0,"phase":"EXTERNALIZE","validated":true}

As you can see:

  • latest known ledger: 26
  • legder 27 could not be validated (node was down)
  • ledger 27 has consensus (from other nodes)
  • ledger 28 and above are validated by the current node

If the messages disappear after you've catched up the history, I think it's not a big deal.

I had the issue because I've forgot to initialize the history (--newhistory local) and the validator had issues writing to disk.

[EDIT] I can also see the message when the history can't be downloaded. For example in a private network, if you don't set up a history location, nodes can't recover. See for instance: https://github.com/stellar/stellar-core/blob/master/docs/history.md#catching-up

This bears repeating: peers never send historical data to one another directly, and they must share access to a common history archive if they're ever to successfully catch up with one another when out of sync.

My local error in Stellar log:

2018-11-03T07:20:22.262 GBTLF [Process WARNING] process 13002 exited 1: cp /stellar-installation/history/vs/history/00/00/bc/history-0000bcff.json /stellar-installation/buckets/tmp/history-589ade37100c739d/stellar-history.json

So the node tries to get its own archive, which has never existed because it was down at this very time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.