stellar-code stuck with Joining SCP

Question

I have a private network running on my local mac laptop. The network has two nodes. Both the nodes were syncing correctly on Friday, March 16, 2018. I shutdown the nodes by ctrl c. Now when I start the nodes it is stuck at "Joining SCP"

stellar-core --c 'info'

{
   "info" : {
      "UNSAFE_QUORUM" : "UNSAFE QUORUM ALLOWED",
      "build" : "v9.2.0rc4-3-g589dcee0",
      "ledger" : {
         "age" : 259612,
         "baseFee" : 100,
         "baseReserve" : 100000000,
         "closeTime" : 1521223082,
         "hash" : "c5ee970107f8a4c050fa3d0291e8dd3bae7593f36a67cf5194ca1f9908396ea4",
         "num" : 9594,
         "version" : 0
      },
      "network" : "My Network ; March 2018",
      "peers" : {
         "authenticated_count" : 1,
         "pending_count" : 0
      },
      "protocol_version" : 9,
      "startedOn" : "2018-03-19T17:41:06Z",
      "state" : "Joining SCP"
   }
}

stellar-core --c 'quorum'

{
   "node" : "self",
   "slots" : {
      "9594" : {
         "agree" : 1,
         "disagree" : [ "two" ],
         "fail_at" : 0,
         "fail_with" : null,
         "hash" : "6d67a3",
         "missing" : null,
         "phase" : "PREPARE",
         "value" : {
            "t" : 2,
            "v" : [ "self", "two" ]
         }
      }
   }
}

In addition I see logs

2018-03-19T11:12:02.545 GAOQY [Overlay WARNING] idle timeout
2018-03-19T11:12:02.546 GAOQY [Overlay INFO] Dropping peer [email protected]:11627
2018-03-19T11:12:02.546 GAOQY [Overlay ERROR] TCPPeer::drop shutdown socket failed: Socket is not connected [TCPPeer.cpp:186]
2018-03-19T11:12:03.586 GAOQY [Overlay INFO] New connected peer 127.0.0.1:0
2018-03-19T11:12:03.588 GAOQY [Overlay INFO] successful handshake with [email protected]:11627

I keep seeing the above error continuously. I see same logs and command output on other node only difference being it lists other node (one)

It seems there is disagreement between the nodes "disagree" : [ "two" ]. Is there a way to have them agree and sync. How can I find the reason of disagreement.

Thanks for all the help and support

[March 21, 2018] I tried again with leaving the servers running same state "Joining SCP". This time I added a additional node to the mix making it 3 nodes. Making the 3rd code as history node collecting all the info. 3rd node is in state "Synced!".

I performed reset on two nodes

stellar-core --newdb
stellar-core --newhist h1
stellar-core --forcescp
stellar-core

I can see the nodes accessing historyarchive/.well-known/stellar-history.json HTTP/1.1" 200 3540 checking the apache access log. State did not not change still "Joining SCP". I have CATCHUP_RECENT=60480 in config. I see following error in log TCP: message size unacceptable: 1213486160 while not authenticated [TCPPeer.cpp:342] Is this TCP error the reason node is not performing the catchup.

Answer 1

I believe this situation is spelled out in the testnet guide.

If you need to restart the network after bringing it down.

Stop all nodes, and do the following on nodes that all have the same last ledger (NB: this set must form a quorum in order to reach consensus):

$ stellar-core --forcescp
$ stellar-core

This will start from the last saved state of each server. After these servers sync you can start the other nodes in the cluster normally and they will catch up to the network.

Answer 2

Okay so had faced this issue with my private network as well. here are the things which I did to fix it.

Test1

1. Started network with 2 nodes. with threshold percent of 100
2. I pulled down the first node and kept the 2nd node running
3. Re-started the 1st node, thinking that it will get synced with the 2nd node but SCP got stuck

Reason:

Since I set the threshold percent to 100, which means both the nodes needed for SCP and when I pulled down the 1st node, threshold did not reach. so set the threshold percent 50

Test2

1. After setting a threshold =50%, restarted both nodes.
2. I pulled down the first node and kept the 2nd node running (this time SCP was happening)
3. Re-started the 1st node, thinking that it will get synced - but this time got stucked in "Joining SCP"

Reason

Quorum was not set properly, I made self and the other node as validators in both of the nodes. Also I had to fix NODE_NAMES key in config file.

Test3

1. After fixing everything, followed the above steps again
2. This time it got stuck into "catching up" state.

Reason

I did not properly set the get of HISTORY. I had to point to the history archive path of the other node2 in node1's configuration file.

This is the config file I created.

Note*: Ignore the bitcoin-related keys in that config file.

Hope this helps!

Answer 3

Same error there bootstrapping a standalone network on GCP.

In our case was caused by the PEER_PORT closed on the firewall.

Remember that Stellar Full Validator have two inbound ports

• 11625 (default PEER_PORT)
• 11626 (default HTTP_PORT)

For me is not clear if Full Validators are using also HTTP_PORT as a part of the consensus but:

[Overlay ERROR] TCPPeer::drop shutdown socket failed: Socket is not connected [TCPPeer.cpp:186]

is telling us that we run into problem on the PEER_PORT socket

[Overlay ERROR] TCP: message size unacceptable: 1213486160 while not authenticated

is telling us that the node receives an HTTP package

For the sake of kownledge (not releated to Stellar), @lovasoa pointed there that

1213486160 is the decimal representation of 0x48 0x54 0x54 0x50 which is HTTP in ascii