Stellar Core node is typically behind SDF by 20 to 100 ledgers

Question

I'm attempting to synch Stellar core (v10.0.0) and most of the time, it is behind https://horizon.stellar.org/ by 20 to 100 ledgers.

To make sure it's not just horizon that's lagging, I check the info of the Stellar node and, although a little more difficult to get a high-level view, I check the logs.

NODE_IS_VALIDATOR is true
NODE_SEED is set
The node has been able to get to the "Synced!" state, but it did not stay there long
I see a lot of "Dropping peer" and "New connected peer" entries in the log
It sometimes pauses in successful handshake with. For example, as I write this, that has been the latest log entry for 20 minutes

I will update this question as I know what else to add to help people diagnose the problem.

My config is below:

LOG_FILE_PATH=""

BUCKET_DIR_PATH="/data/buckets"

DATABASE="postgresql://XXX"

HTTP_PORT=11626

PUBLIC_HTTP_PORT=true

HTTP_MAX_CLIENT=128

NODE_NAMES=[
"GDB5CSAUDVBYBFRBTUWDGANT7BK4NW7UEZWDEQGFS4EYAT6APTY3PWHT bloom-01",
"GCWRN4VKDQIES27TST5FHEKX23JZQZZDXZD4NVZSHIBGJBLOUTF64LMC bloom-02",
"GCUKUUGE43DB7NXWLBLNYYLXH7UF5JHGEEZEEU3VOHOCXMHMOV3CH6JF bloom-03",
"GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH sdf1",
"GCM6QMP3DLRPTAZW2UZPCPX2LF3SXWXKPMP3GKFZBDSF3QZGV2G5QSTK sdf2",
"GABMKJM6I25XI4K7U6XWMULOUQIQ27BCTMLS6BYYSOWKTBUXVRJSXHYQ sdf3",
"GCGWABAQ6OUOVUGWJVPRJ5LWBIWYN3CVOVOZYBNQQGIBRULQHYNGQ7GH cryptomover1",
"GC7MH45NSXXPBLQJRSEVF2DFUVLGGYOJER5FRUNVCYVMXJYJT5LLQJW5 cryptomover2",
"GAXEJOMEVVD5OAAGOOZ4SXTTNR46C4V23XROHWRNVEGOTGOABO3ZVH7Z fairx-us",
"GCKWUQGSVO45ZV3QK7POYL7HMFWDKWJVMFVEGUJKCAEVUITUCTQWFSM6 ibm-au",
"GBUJA3Z5TLAKLI5MEH4TETLXJBQVSVW74MNEKP5UUHTP3IMLNSUPOTVA ibm-br",
"GB2HF2NHRKKFZYFDGD7MUENOYROOEK7SWYV2APYOODP6P7BUJTLILKIL ibm-ca",
"GBJ7T3BTLX2BP3T5Q4256PUF7JMDAB35LLO32QRDYE67TDDMN7H33GGE ibm-hk",
"GCH3O5PTCZVR4G65W3B4XDKWI5V677HQB3QO7CW4YPVYDDFBE2GE7G6V ibm-in",
"GAEEH4TBR7YQQWKJ2FIT57HXZZTMK2BX5LY4POJUYFSEZ7Y2ONHPPTES ibm-it",
"GDRA72H7JWXAXWJKOONQOPH3JKNSH5MQ6BO5K74C3X6FO2G3OG464BPU ibm-no",
"GAENPO2XRTTMAJXDWM3E3GAALNLG4HVMKJ4QF525TR25RI42YPEDULOW ibm-uk",
"GARBCBH4YSHUJLYEPKEPMVYZIJ3ZSQR3QCJ245CWGY64X72JLN4A6RSG ibm-us",
"GDIQKLQVOCD5UD6MUI5D5PTPVX7WTP5TAPP5OBMOLENBBD5KG434KYQ2 stronghold1",
"GA7MREQ7673YDVANF4WBPN7LBQM4BSH4BQUFUTC4YLSSQCQUQTXRVBZN stronghold2",
"GDHV7FL7JP6LUEWWYUOA4C3QAFQ4LZ6OHFBWFIL6IFIS3AOBI2DHV62F stronghold3",
"GBGDBLFKR3LORWOI65LVC7ES5OGZ4GHILEHCBVKPW2PMP2OL43F6B2JJ stronghold4",
"GCBXBCIKCCVUIHAZ5QFWK6CKSX4AESOJ33IQNUE42BP7J66X23TM6WPF stronghold5",
"GC5SXLNAM3C4NMGK2PXK4R34B5GNZ47FYQ24ZIBFDFOCU6D4KBN4POAE satoshipay1",
"GBJQUIXUO4XSNPAUT6ODLZUJRV2NPXYASKUBY4G5MYP3M47PCVI55MNT satoshipay2",
"GAK6Z5UVGUVSEK6PEOCAYJISTT5EJBB34PN3NOLEQG2SUKXRVV2F6HZY satoshipay3",
"GCJCSMSPIWKKPR7WEPIQG63PDF7JGGEENRC33OKVBSPUDIRL6ZZ5M7OO tempo",
"GDNSWA3ZJRQI7EBFLUASFZCMXU3WDEOP24M373XTADCT7UYGWBNFWZSK alchymia",
"GACEYMZ5RGGPV3N7WCZ2VWNLA3PZGBBGBKXP4G3GQ7FTIV5IR5ZZEB2R badbox",
"GDCAQY2Q5VMMI4ZOX35GTL7X43LMP3RA5W4VCP6V2BB6IFOO76GACXZU chris.network1",
"GCXEKMPY6TDNNI2VBWFOEA23V22OZW2INSCGTKNEHIHJDPG2C53PURBG chris.network2",
"GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWA6L7U eno",
"GC3WHI444R4ULETZAECXA6B6PI4OHVWUW2RKZYP56H6MFL5IFAXAOYEW flutterwave",
"GAZMF5YXLMTUTMAXRZ3Z3TOJ5F54OZJGRQAUXDY2VZN3JRYD2MIGVH7B fosha",
"GDBK7LDRYUE6OT4BX4DH7RQVOARBFCVNCP22BV3YF7LQRLANEY5AVAGY jfl",
"GCWJKM4EGTGJUVSWUJDPCQEOEP5LHSOFKSA4HALBTOO4T4H3HCHOM6UX keybase",
"GASN57EFNZWME73BJXYZUTCD34EPX4KIIZQTQDTMBWWVH6JIZJUCBGQX goodx",
"GD63J64WLRF3IOLJZODAVLB4PQNMFFLDBIZCRC4GLTLCUE4ZQMIKPJK6 mcna",
"GCJWQ4C5VQKLAVHT2Y6CA7TIM3VBEH62E7LKTIQQJDBZFLMZPFY3ITTV mystellar.tools",
"GC7WI424OUF6UVYOLVLZD7IOW7M6HTJTIGDX4USFMTAG3763PYOXGN7Q numerotron",
"GAG5HH3VF7WLUR3TRQSJGNX66GSEA5HKH3QPIYREDCT4JXUCD4U43JBJ poiuty",
"GDEP5ASQQT4LKZLK6POEQKPTL7SXWQ66QW3WIRXFN4WXFL5JBG3K5GKQ republic",
"GBB32UXWEXGZUE7H7LUVNNZRT3ZMZ3YH7SP3V5EFBILUVL3NCTSSK3IZ stellarport1",
"GC5A5WKAPZU5ASNMLNCAMLW7CVHMLJJAKHSZZHE2KWGAJHZ4EW6TQ7PB stellarport2",
"GD7FVHL2KUTUYNOJFRUUDJPDRO2MAZJ5KP6EBCU6LKXHYGZDUFBNHXQI umbrel",
"GCEFOV5PD54MX2BS4Q5EQNE6L7JRTDYKEFT7XYBRBWXI3HL47N5U7TSU xavi.ablaza"
]

NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"

PEER_PORT=11625

PREFERRED_PEERS=[
"core-live-a.stellar.org:11625",
"core-live-b.stellar.org:11625",
"core-live-c.stellar.org:11625",
"ohio-1.stellar.stellarport.io:11625",
"ohio-2.stellar.stellarport.io:11625",
"stellar.256kw.com:11625",
"stellar1.tempo.eu.com:11625",
"stellar1.satoshipay.io:11625",
"stellar2.satoshipay.io:11625",
"stellar3.satoshipay.io:11625",
"validator1.stellar.stronghold.co:11625",
"validator2.stellar.stronghold.co:11625",
"validator3.stellar.stronghold.co:11625",
"validator4.stellar.stronghold.co:11625",
"validator5.stellar.stronghold.co:11625",
"au.stellar.ibm.com:11625",
"br.stellar.ibm.com:11625",
"ca.stellar.ibm.com:11625",
"hk.stellar.ibm.com:11625",
"in.stellar.ibm.com:11625",
"it.stellar.ibm.com:11625",
"no.stellar.ibm.com:11625",
"uk.stellar.ibm.com:11625",
"us.stellar.ibm.com:11625"
]

KNOWN_PEERS=[
"stellar2.clic.world:11625",
"stellar.cowriesys.com:11625",
"cryptodealer.hk:11625",
"cryptomover.org:11625",
"cryptomover.net:11625",
"stellar.256kw.com:11625",
"de.stellar.goodx.network:11625",
"au.stellar.ibm.com:11625",
"br.stellar.ibm.com:11625",
"ca.stellar.ibm.com:11625",
"hk.stellar.ibm.com:11625",
"in.stellar.ibm.com:11625",
"it.stellar.ibm.com:11625",
"no.stellar.ibm.com:11625",
"uk.stellar.ibm.com:11625",
"us.stellar.ibm.com:11625",
"us1.stellar.knackforge.com:11625",
"koinbros.org:11625",
"v1.stellar.lobstr.co:11625",
"v2.stellar.lobstr.co:11625",
"localstellarassets.com:11625",
"us.stellar.mobius.network:11625",
"core-live-a.stellar.org:11625",
"core-live-b.stellar.org:11625",
"core-live-c.stellar.org:11625",
"stellar1.satoshipay.io:11625",
"stellar2.satoshipay.io:11625",
"stellar3.satoshipay.io:11625",
"stellar.smartlands.io:11625",
"ohio-1.stellar.stellarport.io:11625",
"ohio-2.stellar.stellarport.io:11625",
"validator1.stellar.stronghold.co:11625",
"validator2.stellar.stronghold.co:11625",
"validator3.stellar.stronghold.co:11625",
"validator4.stellar.stronghold.co:11625",
"validator5.stellar.stronghold.co:11625",
"stellar1.tempo.eu.com:11625",
"ton.money.by.tontinetrust.com:11625",
"hoedic.intrastellar.io:11625",
"stellar0.keybase.io:11625",
"ie.paywith.glass:11625",
"validator.stellar.republic.co:11625",
"v1.stellar.smoove.net:11625",
"v2.stellar.smoove.net:11625",
"stellar.papayame.com:11625"
]

NODE_SEED="X"

NODE_IS_VALIDATOR=true

CATCHUP_RECENT=5000

[HISTORY.bloom-01]
get="/usr/bin/gsutil cp gs://bloomx-stellar-core-history/prod/01/{0} {1}"

[HISTORY.bloom-02]
get="/usr/bin/gsutil cp gs://bloomx-stellar-core-history/prod/02/{0} {1}"
put="/usr/bin/gsutil cp {0} gs://bloomx-stellar-core-history/prod/02/{1}"

[HISTORY.sdf1]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

[HISTORY.sdf2]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}"

[HISTORY.sdf3]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}"

[QUORUM_SET]
THRESHOLD_PERCENT=51
VALIDATORS=[
]

[QUORUM_SET.sdf]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$sdf1",
"$sdf2",
"$sdf3"
]

[QUORUM_SET.ibm]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$ibm-au",
"$ibm-br",
"$ibm-ca",
"$ibm-hk",
"$ibm-in",
"$ibm-it",
"$ibm-no",
"$ibm-uk",
"$ibm-us"
]

[QUORUM_SET.satoshipay]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$satoshipay1",
"$satoshipay2",
"$satoshipay3"
]

[QUORUM_SET.stellarport]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$stellarport1",
"$stellarport2"
]

[QUORUM_SET.stronghold]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$stronghold1",
"$stronghold2",
"$stronghold3",
"$stronghold4",
"$stronghold5"
]

[QUORUM_SET.others]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$umbrel",
"$mystellar.tools",
"$chris.network1",
"$chris.network2",
"$flutterwave",
"$xavi.ablaza",
"$tempo",
"$keybase"
]

Here is the result of the info command:

{
  "info" : {
    "build" : "v10.0.0",
    "history" : {
      "bloom-01" : {
        "failure" : 106,
        "success" : 0
      },
      "bloom-02" : {
        "failure" : 0,
        "success" : 30
      },
      "bloom-03" : {
        "failure" : 116,
        "success" : 0
      },
      "sdf1" : {
        "failure" : 5,
        "success" : 183
      },
      "sdf2" : {
        "failure" : 2,
        "success" : 192
      },
      "sdf3" : {
        "failure" : 7,
        "success" : 159
      }
    },
    "ledger" : {
      "age" : 1385,
      "baseFee" : 100,
      "baseReserve" : 5000000,
      "closeTime" : 1537923144,
      "hash" : "cd909b76ef0eee638546cfe0719c104176c8da0de2fda8424c2de84f652a4a43",
      "maxTxSetSize" : 50,
      "num" : 20170709,
      "version" : 10
    },
    "network" : "Public Global Stellar Network ; September 2015",
    "peers" : {
      "authenticated_count" : 11,
      "pending_count" : 0
    },
    "protocol_version" : 10,
    "quorum" : {
      "20170950" : {
        "agree" : 22,
        "disagree" : 0,
        "fail_at" : 2,
        "hash" : "9ce13b",
        "missing" : 8,
        "phase" : "EXTERNALIZE",
        "validated" : false
      }
    },
    "startedOn" : "2018-09-25T08:37:51Z",
    "state" : "Joining SCP",
    "status" : [
      "Catching up: applying checkpoint 1/4 (25%)",
      "Publishing 1451 queued checkpoints [20077631-20170687]: Succeded: prepare-snapshot"
    ]
  }
}

This is a shot in the dark here, but can you check the CPU usage of your Postgres machine? If it's pegged at 100% you may want to try using the latest horizon and disabling asset stats (see github.com/stellar/go/releases/tag/horizon-v0.14.1) — Paul, Commented Sep 25, 2018 at 0:45
kubectl top pods shows that I'm hardly using the pod's resources. Yes I understand -- I don't even know what info to provide to help you help me :) Thanks! — Ramon Tayag, Commented Sep 25, 2018 at 1:20
I just re-read your comment. I just checked the PG db, and it's 10% to 60%. — Ramon Tayag, Commented Sep 25, 2018 at 8:39
Not sure, generally it looks ok. Just to get it synced I'd try to reduce quorum set to 4-5 nodes: sdf1-2-3, maybe couple more that geographically close to you. Put all of them into preferred peers and maybe disable for now your history archives bloom1-2-3. Just to see if it syncs. After that you can slowly get stuff back — umbrel, Commented Sep 26, 2018 at 10:22

umbrel · Accepted Answer · 2018-09-25 01:01:58Z

1

Something must be wrong with your peers. Try to add/revise section PREFERRED_PEERS in your stellar-core.cfg Here's mine

PREFERRED_PEERS=[
"core-live-c.stellar.org:11625", #sdf_1
"core-live-c.stellar.org:11625", #sdf_3

"ca.stellar.ibm.com:11625", #ibm_ca

"stellar3.satoshipay.io:11625", #satoshipay_3

"stellar.256kw.com:11625", #eno
"stellar1.tempo.eu.com:11625", #tempo_1
]

It's not perfect, but it has nodes, that I need for my quorum set and I don't have any problems with syncing since added that.

The reason it's up to 100 ledgers behind: because every 100 ledgers there is a checkpoint, that you download from archive. But real-time syncing is failing

answered Sep 25, 2018 at 1:01

umbrel

9385 silver badges9 bronze badges

I tried this about an hour ago and let it run. I put exactly yours plus a few more from ibm. Core node is still lagging behind. It's currency in the "Synced!" phase but about 80 ledgers behind. Testing out a few more things.
– Ramon Tayag
Commented Sep 25, 2018 at 3:37
Ok, I'm still about 50 ledgers behind. Updating the question with my config. Hopefully it helps.
– Ramon Tayag
Commented Sep 25, 2018 at 6:05

Add a comment |

Chris · Accepted Answer · 2018-09-26 07:48:35Z

0

Another shot in the dark: Maybe it's a problem of your quorum set.

"quorum" : {
  "20170950" : {
    "agree" : 22,
    "disagree" : 0,
    "fail_at" : 2,
    "hash" : "9ce13b",
    "missing" : 8,
    "phase" : "EXTERNALIZE",
    "validated" : false
  }

You are asking 22 nodes, could be a problem of network latency.

answered Sep 26, 2018 at 7:48

Chris

11 bronze badge

Thanks Chris I'll give that a try. I had been trying to figure out why the authenticated_peers in info goes to 0 sometimes. Will see if this changes anything.
– Ramon Tayag
Commented Sep 26, 2018 at 12:05
I brought the nodes down to 15. authenticated_peers was ok, like core usually is after a restart, but is now back to 0. Will bring down the validators to 5 and try again.
– Ramon Tayag
Commented Sep 26, 2018 at 12:35
I set the validators to 5 but I still see authenticated_peers drop to 0 for considerable periods of time. As I write this, it has been zero for 3+ minutes.
– Ramon Tayag
Commented Sep 26, 2018 at 13:16

Add a comment |

Stack Exchange Network

Stellar Core node is typically behind SDF by 20 to 100 ledgers

2 Answers 2

Your Answer

Not the answer you're looking for? Browse other questions tagged
stellar-core
or ask your own question.

Hot Network Questions

Stellar Core node is typically behind SDF by 20 to 100 ledgers

2 Answers 2

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged stellar-core or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
stellar-core
or ask your own question.