Best practices for Stellar core settings KNOWN_PEERS, PREFERRED_PEERS and PREFERRED_PEER_KEYS

Question

What are the best practices for settings KNOWN_PEERS, PREFERRED_PEERS and PREFERRED_PEER_KEYS in the Stellar Core config? Under which circumstances would I use which?

From https://www.stellar.org/developers/stellar-core/software/admin.html#quorum-and-overlay-network:

It is generally a good idea to give information to your validator on other validators that you rely on. This is achieved by configuring KNOWN_PEERS and PREFERRED_PEERS with the addresses of your dependencies.

Additionally, configuring PREFERRED_PEER_KEYS with the keys from your quorum set might be a good idea to give priority to the nodes that allows you to reach consensus.

Without those settings, your validator depends on other nodes on the network to forward you the right messages, which is typically done as a best effort.

This is a bit unclear to me:

• Should I use KNOWN_PEERS or PREFERRED_PEERS? The docs mention both.
• What exactly constitutes a dependency?
• Why should I use PREFERRED_PEER_KEYS instead of PREFERRED_PEERS for quorum sets? Wouldn't PREFERRED_PEERS be better because it already contains ip:port so my node wouldn't need to resolve public key to ip:port?

Here is additional info from https://github.com/stellar/stellar-core/blob/master/docs/stellar-core_example.cfg:

# PREFERRED_PEERS (list of strings) default is empty
# These are IP:port strings that this server will add to its DB of peers.
# This server will try to always stay connected to the other peers on this list.

and

# PREFERRED_PEER_KEYS (list of strings) default is empty
# These are public key identities that this server will treat as preferred
# when connecting, similar to the PREFERRED_PEERS list.
# can use a name already defined in the .cfg

and

# KNOWN_PEERS (list of strings) default is empty
# These are IP:port strings that this server will add to its DB of peers.
# It will try to connect to these when it is below TARGET_PEER_CONNECTIONS.

Answer 1

How Connections (Overlay) Works Today

Here's how the overlay works today:

• When first starting stellar-core, it stores all nodes in KNOWN_PEERS in the peer database as not preferred (false), and all nodes in PREFERRED_PEERS in the peer database as preferred (true).
• When starting to make connections, a stellar-core node always attempts to connect to its preferred peers in the peer database first. If it hits the TARGET_PEER_CONNECTIONS (includes both inbound and outbound connections), it doesn't attempt to make any more connections.
• If the number of connections (inbound & outbound) is < TARGET_PEER_CONNECTIONS, then it starts making connections to other known nodes in the DB. This can be from pre-populating this persistent DB using KNOWN_PEERS, but it adds more peers via messages from other nodes (that say their list of known peers).
• It keeps doing this every few seconds to keep active connections open to other stellar-core nodes.

There are two other aspects to this:

• PREFERRED_PEER_KEYS - whenever you come across a node to connect to, it will check if it is preferred. If there isn't info on whether its IP is already preferred, and the key (node ID) matches one of the keys in PREFERRED_PEER_KEYS, the node will then be marked as preferred. This is useful if you don't know the exact IP address of a given node, but you still want to mark it as preferred.
• PREFERRED_ONLY - this only allows you to connect to nodes you've marked as preferred - any connections made to non-preferred nodes will drop and back-off.

In Summary/Admin's Perspective

Short Answer

• Put node ID's/public keys of nodes in your quorum set in PREFERRED_PEER_KEYS
• Put IP addresses of nodes you know and trust, or simply want to have priority in connections (potentially duplicating the keys you've already specified in PREFERRED_PEER_KEYS) in PREFERRED_KEYS.
• Put any other node you know of in KNOWN_PEERS that shouldn't get special treatment - this is most important when first starting stellar-core to bootstrap the connection process.

Long Answer

• If you know of nodes you already trust and want to have a preference to connect to them (say nodes in your quorum set), you should put them in PREFERRED_PEERS.
• For bootstrapping stellar-core for the first time, if you have additional nodes that have been forwarded to you but you don't want them to have preferential treatment, you add them to KNOWN_PEERS. As you gain more peers in the database via messages, this configuration point becomes less important - its primary purpose is to bootstrap stellar-core when it's first connecting to the network.
• For nodes in your quorum set that you have the keys for, you should put them in PREFERRED_PEERS_KEYS. This gives you a level of redundancy if you have already put these nodes in PREFERRED_PEERS, because if the IP address ever changes you can still find those nodes, have them marked as preferred, and not break consensus because you can't find them via their new IP.
• PREFERRED_ONLY should only be used if you wish to restrict your connections to those you've explicitly specified as preferred, and to ignore messages from nodes about other nodes to connect to.

Explicit Answers To Questions

Should I use KNOWN_PEERS or PREFERRED_PEERS? The docs mention both.

You should use both - KNOWN_PEERS for bootstrapping for the first time, PREFERRED_PEERS for establishing preferential status for nodes (which is important for nodes in your quorum set).

What exactly constitutes a dependency?

A dependency in this context means a node in your quorum set that you're dependent on to reach consensus.

Why should I use PREFERRED_PEER_KEYS instead of PREFERRED_PEERS for quorum sets? Wouldn't PREFERRED_PEERS be better because it already contains ip:port so my node wouldn't need to resolve public key to ip:port?

Yes, PREFERRED_PEERS is marginally faster, but having the keys as well will make your node more robust to IP address changes. We recommend using both corresponding to the same nodes when possible.

Upcoming Changes

The connection strategy is currently being overhauled. Of note is that incoming and outgoing connections will be counted separately, where TARGET_PEER_CONNECTIONS will limit outbound connections, but currently MAX_PENDING_CONNECTIONS will limit inbound connections. When this gets merged in, we'll update the docs accordingly (otherwise, do ping us about it!).