New node is lagging behind the old one

Question

Hi I was doing some experiment (especially after the news of stellar network going down) in my simple 2 nodes private network and got some confusion. Here are the steps I follow:

1. Configured network with 2 nodes, both of them are validators and are included into each others quorum sets with theshold of 50% like this:

[QUORUM_SET]
THRESHOLD_PERCENT=50
VALIDATORS=[
    "$node01",
    "$node02"
]

2. I stopped the 1st node for some time - the SCP still works with a 2nd node only, since the THRESHOLD_PERCENT was 50 (1 out of 2)
3. Restarted the 1st node - it started catching up with the 2nd node.
4. But what I am seeing is they both are running on different ledger state like the 1st node is trailing 2nd node. I was expecting that after some time they both should work on same slot but 1st one is always lagging behind.

What am I missing in concepts ?

EDIT: 1st node is always in "state" : "Catching up","status" : [ "Catching up: Waiting: catchup" ] state. Also none of transaction works on 1st node.

Answer 1

So fixed this problem. My node1's get in history was not pointing to the node2's archive path. After setting them properly it worked.

Node1's history configuration

[HISTORY.local]
get="cp /tmp/stellar-core/node02/history/vs/{0} {1}"
put="cp {0} /tmp/stellar-core/node01/history/vs/{1}"
mkdir="mkdir -p /tmp/stellar-core/node01/history/vs/{0}"

Node2's history configuration

[HISTORY.local]
get="cp /tmp/stellar-core/node01/history/vs/{0} {1}"
put="cp {0} /tmp/stellar-core/node02/history/vs/{1}"
mkdir="mkdir -p /tmp/stellar-core/node02/history/vs/{0}"