I am trying to set up a private stellar cluster with at least 3 nodes. I am able to setup a 2-nodes cluster but when it comes to add another node, the cluster is broken and only one node is synced.
Here is how I configured my 2-nodes cluster.
I started a docker container on each node by running :
docker run --rm -d -p "80:80" -p "8000:8000" -p "11625:11625" -p "11626:11626" -v "/home/ubuntu/stellar:/opt/stellar" --name private-cluster stellar/quickstart --standalone
I logged into the node on 101, stop stellar-core, recreated core database and updated stellar-core.cfg with :
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
NETWORK_PASSPHRASE="MyStandalone Network ; December 2018"
NODE_SEED="SAOQ42SRGOLSVRGJI2BCX4F5PEI3QA3IWN55GCFHQ5PIMJ2CJQTBNCGQ"
NODE_IS_VALIDATOR=true
PREFERRED_PEERS=["192.168.8.102"]
DATABASE="postgresql://dbname=core host=localhost user=stellar password=mypassword"
COMMANDS=["ll?level=debug"]
FAILURE_SAFETY=0
UNSAFE_QUORUM=true
[QUORUM_SET]
THRESHOLD_PERCENT=50
VALIDATORS=["GCYJMPMSMOINIXGLYA7ID45GJ2N46U7LMHFOTYKWHHNBLAA4K6X4FYGE", "GDMEEIAG6KJQAN5RRQHRAB6BDRDJKKPTTTJDQ4WSKEZITQ7S63QGELN7"]
[HISTORY.vs]
get="cp /opt/stellar/core/history/vs/{0} {1}"
put="./cp-and-set-permissions.sh {0} /opt/stellar/core/history/vs/{1}"
mkdir="./mkdir-and-set-permissions.sh -p /opt/stellar/core/history/vs/{0}"
Then I cleaned stellar db :
sudo -u stellar stellar-core --newdb --conf etc/stellar-core.cfg
Moreover I created the history with :
sudo -u stellar stellar-core --newhist vs --conf etc/stellar-core.cfg
I ran :
sudo -u stellar stellar-core --forcescp --conf etc/stellar-core.cfg
Then started stellar-core :
supervisorctl start stellar-core
For the node on 102, I stopped stellar-core, recreated the core database and edited the configuration file like this :
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
NETWORK_PASSPHRASE="MyStandalone Network ; December 2018"
NODE_SEED="SB6O2IMZJH63X4XUHBKUECMBFYB4HAT32G4ALBGNSVFWJLICGHPRQWQD"
NODE_IS_VALIDATOR=true
PREFERRED_PEERS=["192.168.8.101"]
DATABASE="postgresql://dbname=core host=localhost user=stellar password=mypassword"
COMMANDS=["ll?level=debug"]
FAILURE_SAFETY=0
UNSAFE_QUORUM=true
[QUORUM_SET]
THRESHOLD_PERCENT=50
VALIDATORS=["GCYJMPMSMOINIXGLYA7ID45GJ2N46U7LMHFOTYKWHHNBLAA4K6X4FYGE", "GDMEEIAG6KJQAN5RRQHRAB6BDRDJKKPTTTJDQ4WSKEZITQ7S63QGELN7"]
[HISTORY.vs]
get="cp /opt/stellar/core/history/vs/{0} {1}"
put="./cp-and-set-permissions.sh {0} /opt/stellar/core/history/vs/{1}"
mkdir="./mkdir-and-set-permissions.sh -p /opt/stellar/core/history/vs/{0}"
[HISTORY.h1]
get="curl -sf http://192.168.8.101/{0} -o {1}"
I didn't forcescp on 102 as I wanted it to join the cluster on 101. I just cleaned stellar-core db, created my history archive and started stellar-core.
The node on 101 get synced automatically, whereas the one on 102 went to "Joining SCP", then "Catching Up", then "Getting history" and finally "Synced". I did some transfers and when I checked the root account balance it was updated on both nodes. Thus, I had a successful running 2-nodes cluster.
I tried to add one more node in this cluster following the same pattern but whatever configuration I use nodes are not all synced or neither share the same root account balance.
Any help would be welcome.
THRESHOLD_PERCENT=50
is not a good config. You need something >50%, e.g. 51%.