3

I'm attempting to synch Stellar core (v10.0.0) and most of the time, it is behind https://horizon.stellar.org/ by 20 to 100 ledgers.

To make sure it's not just horizon that's lagging, I check the info of the Stellar node and, although a little more difficult to get a high-level view, I check the logs.

  • NODE_IS_VALIDATOR is true
  • NODE_SEED is set
  • The node has been able to get to the "Synced!" state, but it did not stay there long
  • I see a lot of "Dropping peer" and "New connected peer" entries in the log
  • It sometimes pauses in successful handshake with. For example, as I write this, that has been the latest log entry for 20 minutes

I will update this question as I know what else to add to help people diagnose the problem.

My config is below:

LOG_FILE_PATH=""

BUCKET_DIR_PATH="/data/buckets"

DATABASE="postgresql://XXX"

HTTP_PORT=11626

PUBLIC_HTTP_PORT=true

HTTP_MAX_CLIENT=128

NODE_NAMES=[
"GDB5CSAUDVBYBFRBTUWDGANT7BK4NW7UEZWDEQGFS4EYAT6APTY3PWHT bloom-01",
"GCWRN4VKDQIES27TST5FHEKX23JZQZZDXZD4NVZSHIBGJBLOUTF64LMC bloom-02",
"GCUKUUGE43DB7NXWLBLNYYLXH7UF5JHGEEZEEU3VOHOCXMHMOV3CH6JF bloom-03",
"GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH sdf1",
"GCM6QMP3DLRPTAZW2UZPCPX2LF3SXWXKPMP3GKFZBDSF3QZGV2G5QSTK sdf2",
"GABMKJM6I25XI4K7U6XWMULOUQIQ27BCTMLS6BYYSOWKTBUXVRJSXHYQ sdf3",
"GCGWABAQ6OUOVUGWJVPRJ5LWBIWYN3CVOVOZYBNQQGIBRULQHYNGQ7GH cryptomover1",
"GC7MH45NSXXPBLQJRSEVF2DFUVLGGYOJER5FRUNVCYVMXJYJT5LLQJW5 cryptomover2",
"GAXEJOMEVVD5OAAGOOZ4SXTTNR46C4V23XROHWRNVEGOTGOABO3ZVH7Z fairx-us",
"GCKWUQGSVO45ZV3QK7POYL7HMFWDKWJVMFVEGUJKCAEVUITUCTQWFSM6 ibm-au",
"GBUJA3Z5TLAKLI5MEH4TETLXJBQVSVW74MNEKP5UUHTP3IMLNSUPOTVA ibm-br",
"GB2HF2NHRKKFZYFDGD7MUENOYROOEK7SWYV2APYOODP6P7BUJTLILKIL ibm-ca",
"GBJ7T3BTLX2BP3T5Q4256PUF7JMDAB35LLO32QRDYE67TDDMN7H33GGE ibm-hk",
"GCH3O5PTCZVR4G65W3B4XDKWI5V677HQB3QO7CW4YPVYDDFBE2GE7G6V ibm-in",
"GAEEH4TBR7YQQWKJ2FIT57HXZZTMK2BX5LY4POJUYFSEZ7Y2ONHPPTES ibm-it",
"GDRA72H7JWXAXWJKOONQOPH3JKNSH5MQ6BO5K74C3X6FO2G3OG464BPU ibm-no",
"GAENPO2XRTTMAJXDWM3E3GAALNLG4HVMKJ4QF525TR25RI42YPEDULOW ibm-uk",
"GARBCBH4YSHUJLYEPKEPMVYZIJ3ZSQR3QCJ245CWGY64X72JLN4A6RSG ibm-us",
"GDIQKLQVOCD5UD6MUI5D5PTPVX7WTP5TAPP5OBMOLENBBD5KG434KYQ2 stronghold1",
"GA7MREQ7673YDVANF4WBPN7LBQM4BSH4BQUFUTC4YLSSQCQUQTXRVBZN stronghold2",
"GDHV7FL7JP6LUEWWYUOA4C3QAFQ4LZ6OHFBWFIL6IFIS3AOBI2DHV62F stronghold3",
"GBGDBLFKR3LORWOI65LVC7ES5OGZ4GHILEHCBVKPW2PMP2OL43F6B2JJ stronghold4",
"GCBXBCIKCCVUIHAZ5QFWK6CKSX4AESOJ33IQNUE42BP7J66X23TM6WPF stronghold5",
"GC5SXLNAM3C4NMGK2PXK4R34B5GNZ47FYQ24ZIBFDFOCU6D4KBN4POAE satoshipay1",
"GBJQUIXUO4XSNPAUT6ODLZUJRV2NPXYASKUBY4G5MYP3M47PCVI55MNT satoshipay2",
"GAK6Z5UVGUVSEK6PEOCAYJISTT5EJBB34PN3NOLEQG2SUKXRVV2F6HZY satoshipay3",
"GCJCSMSPIWKKPR7WEPIQG63PDF7JGGEENRC33OKVBSPUDIRL6ZZ5M7OO tempo",
"GDNSWA3ZJRQI7EBFLUASFZCMXU3WDEOP24M373XTADCT7UYGWBNFWZSK alchymia",
"GACEYMZ5RGGPV3N7WCZ2VWNLA3PZGBBGBKXP4G3GQ7FTIV5IR5ZZEB2R badbox",
"GDCAQY2Q5VMMI4ZOX35GTL7X43LMP3RA5W4VCP6V2BB6IFOO76GACXZU chris.network1",
"GCXEKMPY6TDNNI2VBWFOEA23V22OZW2INSCGTKNEHIHJDPG2C53PURBG chris.network2",
"GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWA6L7U eno",
"GC3WHI444R4ULETZAECXA6B6PI4OHVWUW2RKZYP56H6MFL5IFAXAOYEW flutterwave",
"GAZMF5YXLMTUTMAXRZ3Z3TOJ5F54OZJGRQAUXDY2VZN3JRYD2MIGVH7B fosha",
"GDBK7LDRYUE6OT4BX4DH7RQVOARBFCVNCP22BV3YF7LQRLANEY5AVAGY jfl",
"GCWJKM4EGTGJUVSWUJDPCQEOEP5LHSOFKSA4HALBTOO4T4H3HCHOM6UX keybase",
"GASN57EFNZWME73BJXYZUTCD34EPX4KIIZQTQDTMBWWVH6JIZJUCBGQX goodx",
"GD63J64WLRF3IOLJZODAVLB4PQNMFFLDBIZCRC4GLTLCUE4ZQMIKPJK6 mcna",
"GCJWQ4C5VQKLAVHT2Y6CA7TIM3VBEH62E7LKTIQQJDBZFLMZPFY3ITTV mystellar.tools",
"GC7WI424OUF6UVYOLVLZD7IOW7M6HTJTIGDX4USFMTAG3763PYOXGN7Q numerotron",
"GAG5HH3VF7WLUR3TRQSJGNX66GSEA5HKH3QPIYREDCT4JXUCD4U43JBJ poiuty",
"GDEP5ASQQT4LKZLK6POEQKPTL7SXWQ66QW3WIRXFN4WXFL5JBG3K5GKQ republic",
"GBB32UXWEXGZUE7H7LUVNNZRT3ZMZ3YH7SP3V5EFBILUVL3NCTSSK3IZ stellarport1",
"GC5A5WKAPZU5ASNMLNCAMLW7CVHMLJJAKHSZZHE2KWGAJHZ4EW6TQ7PB stellarport2",
"GD7FVHL2KUTUYNOJFRUUDJPDRO2MAZJ5KP6EBCU6LKXHYGZDUFBNHXQI umbrel",
"GCEFOV5PD54MX2BS4Q5EQNE6L7JRTDYKEFT7XYBRBWXI3HL47N5U7TSU xavi.ablaza"
]

NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"

PEER_PORT=11625

PREFERRED_PEERS=[
"core-live-a.stellar.org:11625",
"core-live-b.stellar.org:11625",
"core-live-c.stellar.org:11625",
"ohio-1.stellar.stellarport.io:11625",
"ohio-2.stellar.stellarport.io:11625",
"stellar.256kw.com:11625",
"stellar1.tempo.eu.com:11625",
"stellar1.satoshipay.io:11625",
"stellar2.satoshipay.io:11625",
"stellar3.satoshipay.io:11625",
"validator1.stellar.stronghold.co:11625",
"validator2.stellar.stronghold.co:11625",
"validator3.stellar.stronghold.co:11625",
"validator4.stellar.stronghold.co:11625",
"validator5.stellar.stronghold.co:11625",
"au.stellar.ibm.com:11625",
"br.stellar.ibm.com:11625",
"ca.stellar.ibm.com:11625",
"hk.stellar.ibm.com:11625",
"in.stellar.ibm.com:11625",
"it.stellar.ibm.com:11625",
"no.stellar.ibm.com:11625",
"uk.stellar.ibm.com:11625",
"us.stellar.ibm.com:11625"
]

KNOWN_PEERS=[
"stellar2.clic.world:11625",
"stellar.cowriesys.com:11625",
"cryptodealer.hk:11625",
"cryptomover.org:11625",
"cryptomover.net:11625",
"stellar.256kw.com:11625",
"de.stellar.goodx.network:11625",
"au.stellar.ibm.com:11625",
"br.stellar.ibm.com:11625",
"ca.stellar.ibm.com:11625",
"hk.stellar.ibm.com:11625",
"in.stellar.ibm.com:11625",
"it.stellar.ibm.com:11625",
"no.stellar.ibm.com:11625",
"uk.stellar.ibm.com:11625",
"us.stellar.ibm.com:11625",
"us1.stellar.knackforge.com:11625",
"koinbros.org:11625",
"v1.stellar.lobstr.co:11625",
"v2.stellar.lobstr.co:11625",
"localstellarassets.com:11625",
"us.stellar.mobius.network:11625",
"core-live-a.stellar.org:11625",
"core-live-b.stellar.org:11625",
"core-live-c.stellar.org:11625",
"stellar1.satoshipay.io:11625",
"stellar2.satoshipay.io:11625",
"stellar3.satoshipay.io:11625",
"stellar.smartlands.io:11625",
"ohio-1.stellar.stellarport.io:11625",
"ohio-2.stellar.stellarport.io:11625",
"validator1.stellar.stronghold.co:11625",
"validator2.stellar.stronghold.co:11625",
"validator3.stellar.stronghold.co:11625",
"validator4.stellar.stronghold.co:11625",
"validator5.stellar.stronghold.co:11625",
"stellar1.tempo.eu.com:11625",
"ton.money.by.tontinetrust.com:11625",
"hoedic.intrastellar.io:11625",
"stellar0.keybase.io:11625",
"ie.paywith.glass:11625",
"validator.stellar.republic.co:11625",
"v1.stellar.smoove.net:11625",
"v2.stellar.smoove.net:11625",
"stellar.papayame.com:11625"
]

NODE_SEED="X"

NODE_IS_VALIDATOR=true

CATCHUP_RECENT=5000

[HISTORY.bloom-01]
get="/usr/bin/gsutil cp gs://bloomx-stellar-core-history/prod/01/{0} {1}"

[HISTORY.bloom-02]
get="/usr/bin/gsutil cp gs://bloomx-stellar-core-history/prod/02/{0} {1}"
put="/usr/bin/gsutil cp {0} gs://bloomx-stellar-core-history/prod/02/{1}"

[HISTORY.sdf1]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

[HISTORY.sdf2]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}"

[HISTORY.sdf3]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}"

[QUORUM_SET]
THRESHOLD_PERCENT=51
VALIDATORS=[
]

[QUORUM_SET.sdf]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$sdf1",
"$sdf2",
"$sdf3"
]

[QUORUM_SET.ibm]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$ibm-au",
"$ibm-br",
"$ibm-ca",
"$ibm-hk",
"$ibm-in",
"$ibm-it",
"$ibm-no",
"$ibm-uk",
"$ibm-us"
]

[QUORUM_SET.satoshipay]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$satoshipay1",
"$satoshipay2",
"$satoshipay3"
]

[QUORUM_SET.stellarport]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$stellarport1",
"$stellarport2"
]

[QUORUM_SET.stronghold]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$stronghold1",
"$stronghold2",
"$stronghold3",
"$stronghold4",
"$stronghold5"
]

[QUORUM_SET.others]
THRESHOLD_PERCENT=51
VALIDATORS=[
"$umbrel",
"$mystellar.tools",
"$chris.network1",
"$chris.network2",
"$flutterwave",
"$xavi.ablaza",
"$tempo",
"$keybase"
]

Here is the result of the info command:

{
  "info" : {
    "build" : "v10.0.0",
    "history" : {
      "bloom-01" : {
        "failure" : 106,
        "success" : 0
      },
      "bloom-02" : {
        "failure" : 0,
        "success" : 30
      },
      "bloom-03" : {
        "failure" : 116,
        "success" : 0
      },
      "sdf1" : {
        "failure" : 5,
        "success" : 183
      },
      "sdf2" : {
        "failure" : 2,
        "success" : 192
      },
      "sdf3" : {
        "failure" : 7,
        "success" : 159
      }
    },
    "ledger" : {
      "age" : 1385,
      "baseFee" : 100,
      "baseReserve" : 5000000,
      "closeTime" : 1537923144,
      "hash" : "cd909b76ef0eee638546cfe0719c104176c8da0de2fda8424c2de84f652a4a43",
      "maxTxSetSize" : 50,
      "num" : 20170709,
      "version" : 10
    },
    "network" : "Public Global Stellar Network ; September 2015",
    "peers" : {
      "authenticated_count" : 11,
      "pending_count" : 0
    },
    "protocol_version" : 10,
    "quorum" : {
      "20170950" : {
        "agree" : 22,
        "disagree" : 0,
        "fail_at" : 2,
        "hash" : "9ce13b",
        "missing" : 8,
        "phase" : "EXTERNALIZE",
        "validated" : false
      }
    },
    "startedOn" : "2018-09-25T08:37:51Z",
    "state" : "Joining SCP",
    "status" : [
      "Catching up: applying checkpoint 1/4 (25%)",
      "Publishing 1451 queued checkpoints [20077631-20170687]: Succeded: prepare-snapshot"
    ]
  }
}
  • This is a shot in the dark here, but can you check the CPU usage of your Postgres machine? If it's pegged at 100% you may want to try using the latest horizon and disabling asset stats (see github.com/stellar/go/releases/tag/horizon-v0.14.1) – Paul Sep 25 '18 at 0:45
  • kubectl top pods shows that I'm hardly using the pod's resources. Yes I understand -- I don't even know what info to provide to help you help me :) Thanks! – Ramon Tayag Sep 25 '18 at 1:20
  • I just re-read your comment. I just checked the PG db, and it's 10% to 60%. – Ramon Tayag Sep 25 '18 at 8:39
  • Not sure, generally it looks ok. Just to get it synced I'd try to reduce quorum set to 4-5 nodes: sdf1-2-3, maybe couple more that geographically close to you. Put all of them into preferred peers and maybe disable for now your history archives bloom1-2-3. Just to see if it syncs. After that you can slowly get stuff back – umbrel Sep 26 '18 at 10:22
  • What's the output of peers command? – umbrel Sep 26 '18 at 10:24
1

Something must be wrong with your peers. Try to add/revise section PREFERRED_PEERS in your stellar-core.cfg Here's mine

PREFERRED_PEERS=[
"core-live-c.stellar.org:11625", #sdf_1
"core-live-c.stellar.org:11625", #sdf_3

"ca.stellar.ibm.com:11625", #ibm_ca

"stellar3.satoshipay.io:11625", #satoshipay_3

"stellar.256kw.com:11625", #eno
"stellar1.tempo.eu.com:11625", #tempo_1
]

It's not perfect, but it has nodes, that I need for my quorum set and I don't have any problems with syncing since added that.

The reason it's up to 100 ledgers behind: because every 100 ledgers there is a checkpoint, that you download from archive. But real-time syncing is failing

  • I tried this about an hour ago and let it run. I put exactly yours plus a few more from ibm. Core node is still lagging behind. It's currency in the "Synced!" phase but about 80 ledgers behind. Testing out a few more things. – Ramon Tayag Sep 25 '18 at 3:37
  • Ok, I'm still about 50 ledgers behind. Updating the question with my config. Hopefully it helps. – Ramon Tayag Sep 25 '18 at 6:05
0

Another shot in the dark: Maybe it's a problem of your quorum set.

"quorum" : {
  "20170950" : {
    "agree" : 22,
    "disagree" : 0,
    "fail_at" : 2,
    "hash" : "9ce13b",
    "missing" : 8,
    "phase" : "EXTERNALIZE",
    "validated" : false
  }

You are asking 22 nodes, could be a problem of network latency.

  • Thanks Chris I'll give that a try. I had been trying to figure out why the authenticated_peers in info goes to 0 sometimes. Will see if this changes anything. – Ramon Tayag Sep 26 '18 at 12:05
  • I brought the nodes down to 15. authenticated_peers was ok, like core usually is after a restart, but is now back to 0. Will bring down the validators to 5 and try again. – Ramon Tayag Sep 26 '18 at 12:35
  • I set the validators to 5 but I still see authenticated_peers drop to 0 for considerable periods of time. As I write this, it has been zero for 3+ minutes. – Ramon Tayag Sep 26 '18 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.