2

I have a private network running on my local mac laptop. The network has two nodes. Both the nodes were syncing correctly on Friday, March 16, 2018. I shutdown the nodes by ctrl c. Now when I start the nodes it is stuck at "Joining SCP"

stellar-core --c 'info'

{
   "info" : {
      "UNSAFE_QUORUM" : "UNSAFE QUORUM ALLOWED",
      "build" : "v9.2.0rc4-3-g589dcee0",
      "ledger" : {
         "age" : 259612,
         "baseFee" : 100,
         "baseReserve" : 100000000,
         "closeTime" : 1521223082,
         "hash" : "c5ee970107f8a4c050fa3d0291e8dd3bae7593f36a67cf5194ca1f9908396ea4",
         "num" : 9594,
         "version" : 0
      },
      "network" : "My Network ; March 2018",
      "peers" : {
         "authenticated_count" : 1,
         "pending_count" : 0
      },
      "protocol_version" : 9,
      "startedOn" : "2018-03-19T17:41:06Z",
      "state" : "Joining SCP"
   }
}

stellar-core --c 'quorum'

{
   "node" : "self",
   "slots" : {
      "9594" : {
         "agree" : 1,
         "disagree" : [ "two" ],
         "fail_at" : 0,
         "fail_with" : null,
         "hash" : "6d67a3",
         "missing" : null,
         "phase" : "PREPARE",
         "value" : {
            "t" : 2,
            "v" : [ "self", "two" ]
         }
      }
   }
}

In addition I see logs

2018-03-19T11:12:02.545 GAOQY [Overlay WARNING] idle timeout
2018-03-19T11:12:02.546 GAOQY [Overlay INFO] Dropping peer two@127.0.0.1:11627
2018-03-19T11:12:02.546 GAOQY [Overlay ERROR] TCPPeer::drop shutdown socket failed: Socket is not connected [TCPPeer.cpp:186]
2018-03-19T11:12:03.586 GAOQY [Overlay INFO] New connected peer 127.0.0.1:0
2018-03-19T11:12:03.588 GAOQY [Overlay INFO] successful handshake with two@127.0.0.1:11627

I keep seeing the above error continuously. I see same logs and command output on other node only difference being it lists other node (one)

It seems there is disagreement between the nodes "disagree" : [ "two" ]. Is there a way to have them agree and sync. How can I find the reason of disagreement.

Thanks for all the help and support

[March 21, 2018] I tried again with leaving the servers running same state "Joining SCP". This time I added a additional node to the mix making it 3 nodes. Making the 3rd code as history node collecting all the info. 3rd node is in state "Synced!".

I performed reset on two nodes

stellar-core --newdb
stellar-core --newhist h1
stellar-core --forcescp
stellar-core

I can see the nodes accessing historyarchive/.well-known/stellar-history.json HTTP/1.1" 200 3540 checking the apache access log. State did not not change still "Joining SCP". I have CATCHUP_RECENT=60480 in config. I see following error in log TCP: message size unacceptable: 1213486160 while not authenticated [TCPPeer.cpp:342] Is this TCP error the reason node is not performing the catchup.

1

I believe this situation is spelled out in the testnet guide.

If you need to restart the network after bringing it down.

Stop all nodes, and do the following on nodes that all have the same last ledger (NB: this set must form a quorum in order to reach consensus):

$ stellar-core --forcescp
$ stellar-core

This will start from the last saved state of each server. After these servers sync you can start the other nodes in the cluster normally and they will catch up to the network.

  • Sorry for the delayed response. I am following the above steps to restart my nodes (stellar-core --forcescp, stellar-core). forcescp value is reset to false in DB after restart. If my nodes are idle for sometime (no transactions) it goes to Joining SCP state and there is no way back except reset. I think the error message size unacceptable: 1213486160 while not authenticated [TCPPeer.cpp:342] should be causing this issue – Harmit Mar 28 '18 at 0:11
  • 1
    Yes that issue means one node is not able to complete the handshake with the other... which would definitely cause SCP issues. – Rob Mar 28 '18 at 18:13
  • I need to figure this out why one node is not able to compete the handshake. Is it because 2 sets of (stellar-core, horizon, bridge & compliance), callback server, postgres etc. are running on a laptop. I am planning to deploy individual set on a amazon instance and see how it works. Need to know about the error message size unacceptable: 1213486160 while not authenticated [TCPPeer.cpp:342] – Harmit Mar 28 '18 at 23:25
0

Okay so had faced this issue with my private network as well. here are the things which I did to fix it.

Test1

  1. Started network with 2 nodes. with threshold percent of 100
  2. I pulled down the first node and kept the 2nd node running
  3. Re-started the 1st node, thinking that it will get synced with the 2nd node but SCP got stuck

Reason:

Since I set the threshold percent to 100, which means both the nodes needed for SCP and when I pulled down the 1st node, threshold did not reach. so set the threshold percent 50

Test2

  1. After setting a threshold =50%, restarted both nodes.
  2. I pulled down the first node and kept the 2nd node running (this time SCP was happening)
  3. Re-started the 1st node, thinking that it will get synced - but this time got stucked in "Joining SCP"

Reason

Quorum was not set properly, I made self and the other node as validators in both of the nodes. Also I had to fix NODE_NAMES key in config file.

Test3

  1. After fixing everything, followed the above steps again
  2. This time it got stuck into "catching up" state.

Reason

I did not properly set the get of HISTORY. I had to point to the history archive path of the other node2 in node1's configuration file.

This is the config file I created.

Note*: Ignore the bitcoin-related keys in that config file.

Hope this helps!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.