4

I have a similar question to this but it remains unanswered and there is no documentation to be found when it comes to running a stable and resilient production network.

Setup

I am working on a private network with 3 full validators running. There is a single top level quorum that requires 2 of the 3 validators to agree. Postgres, Horizon and Core each exist in their own Docker container.

History

At the moment I have the history configured so that each validator writes and gets from its own local archive. It can also get from the archives of the other 2 cores in the quorum. This is configured as per the instructions here:

A common configuration is for each peer in a group to have a single history archive that it knows how to write to, while also knowing how to read from all archives in the group.

In my stellar-core.cfg this translates to:

[HISTORY.local]
get="cp /tmp/stellar-core/history/vs/{0} {1}"
put="cp {0} /tmp/stellar-core/history/vs/{1}"
mkdir="mkdir -p /tmp/stellar-core/history/vs/{0}"

[HISTORY.h1]
get="curl http://otherpeer1/history/vs/{0} -o {1}"

[HISTORY.h2]
get="curl http://otherpeer2/history/vs/{0} -o {1}"

Running the network

I startup all of the containers (I'm using Swarm so I just fire up a stack with all of them in) and the network starts running. Stellar does it's thing, we can submit transactions, query horizon etc etc. So far so good.

I then want to simulate node failures or updating docker containers so I restart one of the core containers to see what happens. This is where the trouble begins. The container comes back up and the status is "Joining SCP". When I check the stellar core log I see a lot of these messages:

2019-04-12T10:01:38.753 GBIVX [Process WARNING] process 65 exited 1: gzip -d buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Reached retry limit 0 for gunzip-file buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Scheduling retry #5/32 in 30 sec, for get-and-unzip-remote-file bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed
100   314  100   314    0     0  23311      0 --:--:-- --:--:-- --:--:-- 24153

gzip: buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz: not in gzip format

Uh Oh

I notice the core isn't actually connected to the other peers either and it isn't catching up so I go ahead and issue:

stellar-core http-command "connect?peer=othercore&port=11625"

This causes the status to switch to catching up and I get the countdown to the catchup point. Once the countdown reaches the catchup point though the stellar-core process stops or restarts (I can't determine exactly what) and the container restarts sending it back to the "Joining SCP" stage.

At this point I believe the peer is non-recoverable :(

This is no good for production or my health so I tried some other configurations:

Other configurations

I've looked at the following:

  • All cores read and write to a single history archive - the network starts, I can restart any core or even all cores and the network persists - great! But having a single history archive for the entire network seems a bit risky for production and violates the advice here:

writing to the same archive from different nodes is not supported and will result in undefined behavior, potentially data loss.

  • All cores read and write to multiple history archives. This gives the archives some redundancy but again violates the advice about writing to the same archive. I left this test over night and got some interesting results: the whole network functioned but a missing bucket occurred that couldn't be found in any of the archives - once that happened and I tried restarting a core, it couldn't catchup and the network is toast

  • A single core reads and writes to a history archive. The other 2 cores just read from that history archive. I couldn't even get all 3 cores to sync on starting the network - same error as above with gzip complaining. This isn't going to fly in production anyway since updating the writing node would surely be a suicidal operation for the network (particularly as the 2 non-writers would still be validating ledgers) but was purely testing another configuration option

Question

So I seem to have followed all the guidance published but as soon as I restart a single node that node can never get back into the network. So how do other people do this? How is the Stellar Public network history archives setup so that when a core goes down for maintenance it can catch up again?

  • Two things to fix your situation first: (1) Did you do stellar-core --forcescp before restarting Core process? (2) From your Node-0, could you verify http://otherpeer1/history/vs/XXXX being reachable, and the checkpoint files do exist there? – cesarm Apr 12 at 7:25
  • (1) The core always sets the forcescp flag before starting up - it's part of the docker entrypoint and (2) Yes I see the file in question on the remote host. From the error message it looks to me like the file is downloaded and the part that is failing is the gzip (the final line says it is not in gzip format which is worrying). – Owen Apr 12 at 7:46
  • no further thoughts from me yet...but i'd recommend getting into the docker of Node-0, go to its /tmp/stellar-core/history/vs/YYY directory, and try to wget some files from http://otherpeer1/history/vs/XXX. This ensures the outbound network (from Node-0 docker) working perfectly, and your /tmp/stellar-core/history/vs/ directory is writable. – cesarm Apr 12 at 9:26
  • What confuses me is having all cores read and write to a single history archive produces exactly the behavior I want. I can restart nodes and they come back up and catch up immediately and get back into sync. I can even pull the whole network and bring it back up in sync. However I can't believe having a single history archive (even with frequent backups) is a suitable setup for a production environment... Just can't see what the configuration should actually be – Owen Apr 12 at 9:41
  • I can assure you that the each node can have its own individual history archive. If you have ensured the network config is perfectly fine, then include your stellar-core.cfg and let us have a look into it... The reason why I emphasize on the network config is, many cases are that the firewall blocks certain ports, or the docker ports were simply not correctly mapped. – cesarm Apr 12 at 16:09
3

Author of history & buckets subsystems here.

There are a few steps to work through in understanding the failure / parsing the logs, though if you want the TL;DR version you can cut to the final paragraph.

First, the failure that's occurring is a gunzip command, which usually means you've downloaded non-gzip data and labeled it something.gz, which usually happens when your curl command is run without -f and so it's considering a 404 on fetch as "successful" and writing something like an HTML 404 page to its output file and labeling it .gz. Take a look at the file to confirm, but that is my guess.

Next: why 404? It means the other archives don't have the bucket being requested. Why would it request a bucket the other archives don't have? Sometimes this happens because of timing -- they haven't published such a bucket yet, just wait and retry, they eventually will -- but in this case the key is that this is happening during bucket repair.

Bucket repair is a phase that happens when a node starts up and is missing some of its local buckets. This isn't about accessing history as such, it's just using other nodes' history archives as a fallback once it's failed to find a bucket it needs locally.

It's really trying to (re)assume the state it had at its last commit before you yanked the power cable on the node, and that includes finding all the bucket files referenced by the database (including private buckets it hasn't published to any history archive -- checkpoints are only published once every 64 ledgers).

Why would it fail to find a bucket referenced by the database? This usually happens when you put the buckets in ephemeral storage and then threw it away when you rebooted the node. Did that happen here? Specifically, did the buckets directory get deleted / reinitialized when you restarted the node? Or is it in persistent / mapped volume storage?

There is always the possibility of other bugs in this sort of code, but most of the time I've seen this sort of failure, it's been due to failure to store the buckets directory in a place that survives a node's reboot. This is mentioned (albeit perhaps not with sufficient emphasis) in the documentation here: https://www.stellar.org/developers/stellar-core/software/admin.html#runtime-information-start-and-stop

Stellar-core can also be packaged in a container system such as Docker, so long as BUCKET_DIR_PATH and the database are stored on persistent volumes

  • I was missing this. I've added the BUCKET_DIR_PATH to my stellar-core.cfg and made sure to persist it between container restarts and everything is working as expected. One question about this - is it useful to have this directory available to other cores or is it only specific to that particular core? Or to phrase the question slightly differently should I serve it over HTTP in the same way I serve the history archive or it is not used by other cores? – Owen Apr 18 at 3:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.