1

I have a private network based on Stellar core v.10 with 3 nodes and 66% 2/3 consensus, after 6 month successfully worked this network crash with message:

insufficient buffer space in xdr_generic_get.

After that I upgrade core till v12.4 Stellar-core and my network worked 10 days and I got similar error

2020-05-01T19:39:17.821 GCEV7 [SCP FATAL] Exception processing SCP messages at 3670016, envelope: {ENV@self |  i: 3670016 | NOMINATE | D: 6bb10d | X: {'[ txH: 34963f, ct: 1588351157, upgrades: [ ] ]'} | Y: {} } [Slot.cpp:152]
2020-05-01T19:39:17.821 GCEV7 [SCP FATAL] Please report this bug along with this log file if this was not expected [Slot.cpp:155]
2020-05-01T19:39:17.821 GCEV7 [default FATAL] Got an exception: Error merging bucket curr=feb70b with snap=000000: insufficient buffer space in xdr_generic_get. There may be a problem with the local filesystem. Ensure that there is enough space to perform that operation and that disc is behaving correctly. [ApplicationUtils.cpp:90]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error merging bucket curr=feb70b with snap=000000: insufficient buffer space in xdr_generic_get. There may be a problem with the local filesystem. Ensure that there is enough space to perform that operation and that disc is behaving correctly.
Signal: SIGABRT (Aborted)

Process finished with exit code 1

I have no idea why this issue was appear and stellar core did not restart. I think problem with BucketList merging that I found in source code. I found some similar issues on github #2159, #2157, #2218 but these issues was solved in v11.1 of Stellar-core.

I would be glad and very grateful to hear opinions on this matter and provide more detailed information for the Stellar developer community.

Improve this question

1 Answer 1

Reset to default
1

Without access to your private network it can be a little tricky to debug.

This seems likely to be a problem with a corrupt bucket file, which can happen if there's an error or data loss during XDR IO. Bucket files are kept on the node's local disk as well as being published to archives. In versions before 11.4 we failed to call fsync() properly during bucket-write, which could lead to corrupt buckets when a power loss or node-reset occurred immediately after a write. This was fixed in 11.4 but a corrupt bucket file created by an earlier version (eg. v10) might remain corrupt even after an upgrade.

It's possible either or both copies of your bucket file -- on the node and/or in your archive -- might be corrupt. It's worth figuring out which case you're seeing. Do you see this problem if you start a clean node, without local state, synchronizing itself by loading state from one of your archives?

Other possibilities include a bad disk or filesystem on the node, or a bug in stellar-core we're not yet aware of.

Improve this answer
2
  • Thank you very much for the comprehensive answer Graydon, I tried on your advice to launch a clean node and unfortunately it also fell from all three archives. I found this file in three archives and tried to dump it with the dump-hdr command, everything ended with an error: [default FATAL] Got an exception: insufficient buffer space in xdr_generic_get Is there any way to repair a damaged file, or maybe generate something to replace in the history archive? Thank you in advance for your response!
    – Bi0Ac1d
    Commented May 5, 2020 at 17:24
  • Unfortunately by the time you have a corrupt bucket file, there is not much you can do; it's the canonical "master copy" of the data, from which hashes are calculated. The best possibility I can picture is to discard the portion of history that includes the corrupt bucket file. That is: if a corrupt bucket entered your history at ledger N, then all of history before N is probably OK, so you might be able to restart a network from a history that ends at ledger N-1, and continue from there. If you like I can help you do that experiment. I'm also worried about how the corruption occurred!
    – Graydon Hoare
    Commented May 6, 2020 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.