I have a similar question to this but it remains unanswered and there is no documentation to be found when it comes to running a stable and resilient production network.
Setup
I am working on a private network with 3 full validators running. There is a single top level quorum that requires 2 of the 3 validators to agree. Postgres, Horizon and Core each exist in their own Docker container.
History
At the moment I have the history configured so that each validator writes and gets from its own local archive. It can also get from the archives of the other 2 cores in the quorum. This is configured as per the instructions here:
A common configuration is for each peer in a group to have a single history archive that it knows how to write to, while also knowing how to read from all archives in the group.
In my stellar-core.cfg this translates to:
[HISTORY.local]
get="cp /tmp/stellar-core/history/vs/{0} {1}"
put="cp {0} /tmp/stellar-core/history/vs/{1}"
mkdir="mkdir -p /tmp/stellar-core/history/vs/{0}"
[HISTORY.h1]
get="curl http://otherpeer1/history/vs/{0} -o {1}"
[HISTORY.h2]
get="curl http://otherpeer2/history/vs/{0} -o {1}"
Running the network
I startup all of the containers (I'm using Swarm so I just fire up a stack with all of them in) and the network starts running. Stellar does it's thing, we can submit transactions, query horizon etc etc. So far so good.
I then want to simulate node failures or updating docker containers so I restart one of the core containers to see what happens. This is where the trouble begins. The container comes back up and the status is "Joining SCP". When I check the stellar core log I see a lot of these messages:
2019-04-12T10:01:38.753 GBIVX [Process WARNING] process 65 exited 1: gzip -d buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Reached retry limit 0 for gunzip-file buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
2019-04-12T10:01:38.753 GBIVX [Work WARNING] Scheduling retry #5/32 in 30 sec, for get-and-unzip-remote-file bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 314 100 314 0 0 23311 0 --:--:-- --:--:-- --:--:-- 24153
gzip: buckets/tmp/repair-buckets-c4d4750e00c0e5d1/bucket/45/10/c9/bucket-4510c94b7119c043c67d598386f270c9db6c76ee7f3c16967abdb523ec455353.xdr.gz: not in gzip format
Uh Oh
I notice the core isn't actually connected to the other peers either and it isn't catching up so I go ahead and issue:
stellar-core http-command "connect?peer=othercore&port=11625"
This causes the status to switch to catching up and I get the countdown to the catchup point. Once the countdown reaches the catchup point though the stellar-core process stops or restarts (I can't determine exactly what) and the container restarts sending it back to the "Joining SCP" stage.
At this point I believe the peer is non-recoverable :(
This is no good for production or my health so I tried some other configurations:
Other configurations
I've looked at the following:
- All cores read and write to a single history archive - the network starts, I can restart any core or even all cores and the network persists - great! But having a single history archive for the entire network seems a bit risky for production and violates the advice here:
writing to the same archive from different nodes is not supported and will result in undefined behavior, potentially data loss.
All cores read and write to multiple history archives. This gives the archives some redundancy but again violates the advice about writing to the same archive. I left this test over night and got some interesting results: the whole network functioned but a missing bucket occurred that couldn't be found in any of the archives - once that happened and I tried restarting a core, it couldn't catchup and the network is toast
A single core reads and writes to a history archive. The other 2 cores just read from that history archive. I couldn't even get all 3 cores to sync on starting the network - same error as above with gzip complaining. This isn't going to fly in production anyway since updating the writing node would surely be a suicidal operation for the network (particularly as the 2 non-writers would still be validating ledgers) but was purely testing another configuration option
Question
So I seem to have followed all the guidance published but as soon as I restart a single node that node can never get back into the network. So how do other people do this? How is the Stellar Public network history archives setup so that when a core goes down for maintenance it can catch up again?
stellar-core --forcescp
before restarting Core process? (2) From your Node-0, could you verifyhttp://otherpeer1/history/vs/XXXX
being reachable, and the checkpoint files do exist there?/tmp/stellar-core/history/vs/YYY
directory, and try towget
some files fromhttp://otherpeer1/history/vs/XXX
. This ensures the outbound network (from Node-0 docker) working perfectly, and your/tmp/stellar-core/history/vs/
directory is writable.stellar-core.cfg
and let us have a look into it... The reason why I emphasize on the network config is, many cases are that the firewall blocks certain ports, or the docker ports were simply not correctly mapped.