Understanding how nodes catch up

Question

I have set up a private network of 4 nodes.

Out of 4 nodes I am writing history in one node only.

Here is the config of node1:

[HISTORY.vs]
get="cp /tmp/node1/sweta/history/vs/{0} {1}"

put="cp {0} /tmp/node1/sweta/history/vs/{1}"

mkdir="mkdir -p /tmp/node1/sweta/history/vs/{0}"

In other nodes I am getting history from node 1. Here is the history config of the other nodes:

[HISTORY.vs]
get="cp /tmp/node1/sweta/history/vs/{0} {1}"

Note: All nodes are on the same ec2 instance.

My query is if node1 due to some reason stops, then how will the network will come in sync?

Answer 1

Your other nodes will run fine until they get out of sync (for whatever reason), then when trying to catch up by accessing the incomplete/broken archive they will get stuck forever because there will be a gap of missing ledgers between the last one accessable from archive and the first one when the node got out of sync. In theory it might be possible to recover an archive from ./buckets data from some of the remaining synced nodes or rollback your ledger to the last working point from your archive but in any case it will end up in a huge mess that you'll want to avoid at any cost.

Side notes (fine for your dev setup but anyways):

• /tmp is not a good location for an archive
• don't rely on a single archiver
• don't run your whole network on a single host/instance