How to configure the quorum set when some non-SDF nodes are not validating?

Question

After checking that my stellar-core node was hanging with status Joining SCP, and seeing in the debug messages that the SCP was receiving all the envelopes normally, I realized that the problem was the quorum:

"16341117" : {
    "agree" : 3,
    "disagree" : null,
    "fail_at" : 0,
    "fail_with" : null,
    "hash" : "cb9b4d",
    "missing" : [ "ibm_brazil", "ibm_us", "fairx_us" ],
    "phase" : "CONFIRM",
    "value" : {
       "t" : 5,
       "v" : [
          "sdf_watcher1",
          "sdf_watcher2",
          "sdf_watcher3",
          "ibm_brazil",
          "ibm_us",
          "fairx_us"
       ]
    }
 }

These nodes (ibm_brazil, ibm_us, and fairx_us) were online on the dashboard, my node was connected to them and a few days ago they were working normally.

After changing the quorum set to only the 3 SDF nodes, it worked. But what is the proper way to configure the quorum sets so that something like this doesn't happen? Is there a way to test if a specific node is validating?

Answer 1

What you probably saw is the result of DDoS countermeasures in the Stellar network (we’re working to make this less aggressive while preserving safety): the way to test that a validator is working (or simply to use it) is done by adding that validator’s ip & port to your “preferred peers” setting (or at a minimum to a validator that you know is connected to your target validator).

In your example: the IBM validators are validating but you didn’t have at the time a good path to them, all you’d need to add all IBM validator keys is to be connected to any one of them (as they are likely connected to each other).

Answer 2

I would recommend taking FairX and IBM off your validator list as they don't have live users on the public network and they don't run operational businesses from these validators yet. Because of this, they could be down and they don't realize it.

I'd recommend the Stronghold and Stellarport validator list as they are live and can catch validator issues more quickly as they have the user base to detect if a transaction is slow.

https://github.com/stellar/docs/blob/master/validators.md