History Archive Configuration for Private Networks

Question

I have a similar question to this but it remains unanswered and there is no documentation to be found when it comes to running a stable and resilient production network.

Setup

I am working on a private network with 3 full validators running. There is a single top level quorum that requires 2 of the 3 validators to agree. Postgres, Horizon and Core each exist in their own Docker container.

History

At the moment I have the history configured so that each validator writes and gets from its own local archive. It can also get from the archives of the other 2 cores in the quorum. This is configured as per the instructions here:

A common configuration is for each peer in a group to have a single history archive that it knows how to write to, while also knowing how to read from all archives in the group.

In my stellar-core.cfg this translates to:

[HISTORY.local]
get="cp /tmp/stellar-core/history/vs/{0} {1}"
put="cp {0} /tmp/stellar-core/history/vs/{1}"
mkdir="mkdir -p /tmp/stellar-core/history/vs/{0}"

[HISTORY.h1]
get="curl http://otherpeer1/history/vs/{0} -o {1}"

[HISTORY.h2]
get="curl http://otherpeer2/history/vs/{0} -o {1}"

Running the network

I startup all of the containers (I'm using Swarm so I just fire up a stack with all of them in) and the network starts running. Stellar does it's thing, we can submit transactions, query horizon etc etc. So far so good.

I then want to simulate node failures or updating docker containers so I restart one of the core containers to see what happens. This is where the trouble begins. The container comes back up and the status is "Joining SCP". When I check the stellar core log I see a lot of these messages:

2019-04-12T10:01:38.753 GBIVX [Process WARNING] process 65 exited 1: gzip -d buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Reached retry limit 0 for gunzip-file buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Scheduling retry #5/32 in 30 sec, for get-and-unzip-remote-file bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100   314  100   314    0     0  23311      0 --:--:-- --:--:-- --:--:-- 24153

gzip: buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz: not in gzip format

Uh Oh

I notice the core isn't actually connected to the other peers either and it isn't catching up so I go ahead and issue:

stellar-core http-command "connect?peer=othercore&port=11625"

This causes the status to switch to catching up and I get the countdown to the catchup point. Once the countdown reaches the catchup point though the stellar-core process stops or restarts (I can't determine exactly what) and the container restarts sending it back to the "Joining SCP" stage.

At this point I believe the peer is non-recoverable :(

This is no good for production or my health so I tried some other configurations:

Other configurations

I've looked at the following:

• All cores read and write to a single history archive - the network starts, I can restart any core or even all cores and the network persists - great! But having a single history archive for the entire network seems a bit risky for production and violates the advice here:

writing to the same archive from different nodes is not supported and will result in undefined behavior, potentially data loss.

• All cores read and write to multiple history archives. This gives the archives some redundancy but again violates the advice about writing to the same archive. I left this test over night and got some interesting results: the whole network functioned but a missing bucket occurred that couldn't be found in any of the archives - once that happened and I tried restarting a core, it couldn't catchup and the network is toast

• A single core reads and writes to a history archive. The other 2 cores just read from that history archive. I couldn't even get all 3 cores to sync on starting the network - same error as above with gzip complaining. This isn't going to fly in production anyway since updating the writing node would surely be a suicidal operation for the network (particularly as the 2 non-writers would still be validating ledgers) but was purely testing another configuration option

Question

So I seem to have followed all the guidance published but as soon as I restart a single node that node can never get back into the network. So how do other people do this? How is the Stellar Public network history archives setup so that when a core goes down for maintenance it can catch up again?

Answer 1

Author of history & buckets subsystems here.

There are a few steps to work through in understanding the failure / parsing the logs, though if you want the TL;DR version you can cut to the final paragraph.

First, the failure that's occurring is a gunzip command, which usually means you've downloaded non-gzip data and labeled it something.gz, which usually happens when your curl command is run without -f and so it's considering a 404 on fetch as "successful" and writing something like an HTML 404 page to its output file and labeling it .gz. Take a look at the file to confirm, but that is my guess.

Next: why 404? It means the other archives don't have the bucket being requested. Why would it request a bucket the other archives don't have? Sometimes this happens because of timing -- they haven't published such a bucket yet, just wait and retry, they eventually will -- but in this case the key is that this is happening during bucket repair.

Bucket repair is a phase that happens when a node starts up and is missing some of its local buckets. This isn't about accessing history as such, it's just using other nodes' history archives as a fallback once it's failed to find a bucket it needs locally.

It's really trying to (re)assume the state it had at its last commit before you yanked the power cable on the node, and that includes finding all the bucket files referenced by the database (including private buckets it hasn't published to any history archive -- checkpoints are only published once every 64 ledgers).

Why would it fail to find a bucket referenced by the database? This usually happens when you put the buckets in ephemeral storage and then threw it away when you rebooted the node. Did that happen here? Specifically, did the buckets directory get deleted / reinitialized when you restarted the node? Or is it in persistent / mapped volume storage?

There is always the possibility of other bugs in this sort of code, but most of the time I've seen this sort of failure, it's been due to failure to store the buckets directory in a place that survives a node's reboot. This is mentioned (albeit perhaps not with sufficient emphasis) in the documentation here: https://www.stellar.org/developers/stellar-core/software/admin.html#runtime-information-start-and-stop

Stellar-core can also be packaged in a container system such as Docker, so long as BUCKET_DIR_PATH and the database are stored on persistent volumes