Huge temporary bucket files created - and then destroyed on all cores. Is this normal?

Question

I'm maintaining a network of Stellar nodes with quite a lot of traffic, for quite some time now (almost half a year). Everything has been quite stable so far, but last week I noticed that all the core apps started exhibiting a new behaviour with respect to disc space: huge (~5G) temporary bucket files are being created and then deleted (in '/stellar-core/volumes/stellar-core/buckets/tmp/')

is this normal? all my cores are performing in the same way: every ~10 minutes, they use up an additional 5Gb of disc space and then release it. I suspect this started happening now because the network now uses lower levels of buckets (in the BucketList). There's no adverse effect - everything works just fine otherwise. Would love to know why this is happening and especially to know that its 'normal'.

Answer 1

author of the bucket and history-publishing subsystems here!

It does sound like the root cause is intertwined with the backing-up history publishing queue. There's a chance this has to do with configuration on your side, so I'd be curious if you could share a log of the publish failures and/or the history archive configuration lines from your stellar-core.cfg. But there's also a chance one or more bugs in stellar-core is causing this (see below).

As far as the cause of the space spikes:

Publishing a checkpoint does require capturing and uploading buckets (those representing the state of the ledger at the time the checkpoint is captured).

In general any bucket is constructed by merging two existing buckets, and in the process of performing such a merge, a new bucket of size similar to the sum of the inputs will be created. If that new bucket happens to be redundant (same content) as an existing bucket, it may be immediately discarded; but this can only be detected by comparing the hash of the bucket at the end of its construction. So "forming new redundant buckets" is certainly possible as a cause of a space-spike.

As far as the cause of repeated spikes / backing up queues:

The process that uploads to a history archive tries to resolve all merges that were pending at the snapshot time before it does an upload. A snapshot is taken on the fly as ledgers are closing, and merges for future buckets may not be complete at the time it's taken. So my guess is that you're in a loop where the history archive is trying to resolve the pending merges in the snapshot order to publish it (producing a space spike) and then failing to publish and re-queueing the snapshot.

Here's where this might be caused by bugs in stellar-core: first, it's possible that we could or should rewrite a snapshot the moment it's resolved and before we try to upload, so that we don't re-perform a merge when re-trying a publication. And second, it's even possible that the publication is failing because of the slow merge, for example there might be a timeout firing during the merge resolve step that decides that publishing as a whole has failed.

I've opened a bug to investigate both possibilities: https://github.com/stellar/stellar-core/issues/2037

Thanks for the report!

Answer 2

We've gained some limited understanding of this issue, although we dont have the entire picture yet.

This entire thing revolves around having a large queue of unpublished checkpoints. It seems that having such a queue (1k in our case) causes the core to create these large files. In addition, it seems that this extra processing inhibits the core from publishing more checkpoints thus making it worse, so to speak. Once the queue was eliminated (by forcing the core app to attempt to publish every 10 seconds), everything went back to normal.

We dont know yet what was the original cause of failure (==what caused the queue to begin with). I'll update more when I find out.