5

I'm maintaining a network of Stellar nodes with quite a lot of traffic, for quite some time now (almost half a year). Everything has been quite stable so far, but last week I noticed that all the core apps started exhibiting a new behaviour with respect to disc space: huge (~5G) temporary bucket files are being created and then deleted (in '/stellar-core/volumes/stellar-core/buckets/tmp/')

is this normal? all my cores are performing in the same way: every ~10 minutes, they use up an additional 5Gb of disc space and then release it. I suspect this started happening now because the network now uses lower levels of buckets (in the BucketList). There's no adverse effect - everything works just fine otherwise. Would love to know why this is happening and especially to know that its 'normal'.

  • could you post your stellar-core.cfg? suspect that might be related to this config: github.com/stellar/stellar-core/blob/master/docs/… (although, according to my experience, this only affects database size) – cesarm Mar 25 at 7:26
  • @cesarm - I dont have any cache configuration in my config so I guess I'm getting the default values. there's nothing there besides the vanilla stuff (other nodes, history buckets, etc). – FuzzyAmi Mar 25 at 8:07
  • Not much idea about it then...maybe post your stellar-core.cfg and let us see if anything looks suspicious... – cesarm Mar 25 at 9:07
  • also, any suggestions on how many subfolders/files under the directory? For my node, there is only an empty buckets/tmp/bucket-20ee242f1c9cde92/ and 2 files local-stellar-history.json & stellar-history.json in buckets/tmp/history-2beb38be0b74637f/ – cesarm Mar 25 at 9:17
4

author of the bucket and history-publishing subsystems here!

It does sound like the root cause is intertwined with the backing-up history publishing queue. There's a chance this has to do with configuration on your side, so I'd be curious if you could share a log of the publish failures and/or the history archive configuration lines from your stellar-core.cfg. But there's also a chance one or more bugs in stellar-core is causing this (see below).

As far as the cause of the space spikes:

Publishing a checkpoint does require capturing and uploading buckets (those representing the state of the ledger at the time the checkpoint is captured).

In general any bucket is constructed by merging two existing buckets, and in the process of performing such a merge, a new bucket of size similar to the sum of the inputs will be created. If that new bucket happens to be redundant (same content) as an existing bucket, it may be immediately discarded; but this can only be detected by comparing the hash of the bucket at the end of its construction. So "forming new redundant buckets" is certainly possible as a cause of a space-spike.

As far as the cause of repeated spikes / backing up queues:

The process that uploads to a history archive tries to resolve all merges that were pending at the snapshot time before it does an upload. A snapshot is taken on the fly as ledgers are closing, and merges for future buckets may not be complete at the time it's taken. So my guess is that you're in a loop where the history archive is trying to resolve the pending merges in the snapshot order to publish it (producing a space spike) and then failing to publish and re-queueing the snapshot.

Here's where this might be caused by bugs in stellar-core: first, it's possible that we could or should rewrite a snapshot the moment it's resolved and before we try to upload, so that we don't re-perform a merge when re-trying a publication. And second, it's even possible that the publication is failing because of the slow merge, for example there might be a timeout firing during the merge resolve step that decides that publishing as a whole has failed.

I've opened a bug to investigate both possibilities: https://github.com/stellar/stellar-core/issues/2037

Thanks for the report!

  • 1
    I should also mention: bucket files are going to get a lot smaller in version 11 of the protocol; there was essentially a missing element of the compaction algorithm causing runaway tombstone accumulation in previous versions, so we should get a couple orders of magnitude relief from anything related to bucket operations reasonably soon. But this is worth fixing regardless! – Graydon Hoare Apr 5 at 18:01
  • I'm thrilled to see your response here. As noted in github, there's some more info here: github.com/stellar/stellar-core/issues/1746 – FuzzyAmi Apr 7 at 5:26
0

We've gained some limited understanding of this issue, although we dont have the entire picture yet.

This entire thing revolves around having a large queue of unpublished checkpoints. It seems that having such a queue (1k in our case) causes the core to create these large files. In addition, it seems that this extra processing inhibits the core from publishing more checkpoints thus making it worse, so to speak. Once the queue was eliminated (by forcing the core app to attempt to publish every 10 seconds), everything went back to normal.

We dont know yet what was the original cause of failure (==what caused the queue to begin with). I'll update more when I find out.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.