0

I have a problem with two stellar-core installations that were installed by the stellar quickstart image (https://hub.docker.com/r/stellar/quickstart/). I have two running testnet-instances of it at different hosters, started more than a month apart. The earlier one froze after some time of working so I just restarted it. Some time afterwards I deployed the second node. Both worked fine for some time but now both stellar-core instances are frozen at the same ledger 1187897. Again, they run on different hosters/environments so I have no clue what happened here. Can anyone relate?

This is as much data as I could obtain. Note that the configs only slightly differ in their "CATCHUP_RECENT" setting. I am unable to find any logfile, maybe this is related to LOG_FILE_PATH="". Stdout is logged but does not contain any useful messages at all (no stellar-core output).

Instance 1 data

  • cat /opt/stellar/core/etc/stellar-core.cfg 
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
LOG_FILE_PATH=""

NETWORK_PASSPHRASE="Test SDF Network ; September 2015"

KNOWN_PEERS=[
"core-testnet1.stellar.org",
"core-testnet2.stellar.org",
"core-testnet3.stellar.org"]

DATABASE="postgresql://dbname=core host=XXX user=XXX password=XXX"
UNSAFE_QUORUM=true
FAILURE_SAFETY=1
CATCHUP_RECENT=208640

#The public keys of the Stellar testnet servers
[QUORUM_SET]
THRESHOLD_PERCENT=51 # rounded up -> 2 nodes out of 3
VALIDATORS=[
"GDKXE2OZMJIPOSLNA6N6F2BVCI3O777I2OOC4BV7VOYUEHYX7RTRYA7Y  sdf1",
"GCUCJTIYXSOXKBSNFGNFWW5MUQ54HKRPGJUTQFJ5RQXZXNOLNXYDHRAP  sdf2",
"GC2V2EFSXN6SQTWVYA5EPJPBWWIMSD2XQNKUOHGEKB535AQE2I6IXV2Z  sdf3"]

[HISTORY.cache]
get="cp /opt/stellar/history-cache/{0} {1}"

#The history store of the Stellar testnet
[HISTORY.h1]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_001/{0} -o {1}"

[HISTORY.h2]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_002/{0} -o {1}"

[HISTORY.h3]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_003/{0} -o {1}"
  • stellar-core http-command 'info'
2019-10-24T08:12:52.307 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:12:52.312 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1208
Content-Type: application/json

2019-10-24T08:12:52.320 GAVEF [default INFO] {
   "info" : {
      "build" : "v11.3.0rc1",
      "history_failure_rate" : "0.25",
      "ledger" : {
         "age" : 835969,
         "baseFee" : 100,
         "baseReserve" : 5000000,
         "closeTime" : 1571068803,
         "hash" : "e70b405e62cf125668422aa7a59bfcfc2910cfbba7d1eda9e1755567ba95712d",
         "maxTxSetSize" : 100,
         "num" : 1187897,
         "version" : 11
      },
      "network" : "Test SDF Network ; September 2015",
      "peers" : {
         "authenticated_count" : 8,
         "pending_count" : 0
      },
      "protocol_version" : 11,
      "quorum" : {
         "node" : "GD65K",
         "qset" : {
            "agree" : 3,
            "delayed" : 0,
            "disagree" : 0,
            "fail_at" : 2,
            "hash" : "59d361",
            "ledger" : 1341320,
            "missing" : 0,
            "phase" : "EXTERNALIZE"
         },
         "transitive" : {
            "critical" : null,
            "intersection" : true,
            "last_check_ledger" : 1329406,
            "node_count" : 4
         }
      },
      "startedOn" : "2019-10-23T14:05:53Z",
      "state" : "Catching up",
      "status" : [ "Catching up: Running: catchup" ]
   }
}
  • stellar-core http-command 'peers'
2019-10-24T08:24:40.910 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:24:40.914 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1732
Content-Type: application/json

2019-10-24T08:24:40.920 GAGNI [default INFO] {
   "authenticated_peers" : {
      "inbound" : null,
      "outbound" : [
         {
            "address" : "13.114.104.43:11625",
            "id" : "GAEPM",
            "olver" : 9,
            "ver" : "stellar-core 11.1.0 (324c1bd61b0e9bada63e0d696d799421b00a7950)"
         },
         {
            "address" : "54.36.166.103:11625",
            "id" : "GAGMH",
            "olver" : 9,
            "ver" : "v11.0.0"
         },
         {
            "address" : "18.140.91.137:11625",
            "id" : "GBAVU",
            "olver" : 9,
            "ver" : "stellar-core 11.1.0 (324c1bd61b0e9bada63e0d696d799421b00a7950)"
         },
         {
            "address" : "157.245.72.119:11625",
            "id" : "GBDU7",
            "olver" : 9,
            "ver" : "stellar-core 11.4.0 (38f51d60bd7aa473edea6038aaf13c255d79c14d)"
         },
         {
            "address" : "34.92.72.55:11625",
            "id" : "GCNDW",
            "olver" : 10,
            "ver" : "stellar-core 12.0.0 (de0b21c0ecc367b6883693bcbf53010941dcd5e6)"
         },
         {
            "address" : "18.207.213.63:11625",
            "id" : "sdf2",
            "olver" : 10,
            "ver" : "stellar-core 12.1.0~rc2 (8afe57913a08deffa247d7b5f837e0b28a54b864)"
         },
         {
            "address" : "152.32.170.134:11625",
            "id" : "GDXSR",
            "olver" : 10,
            "ver" : "stellar-core 12.0.0 (de0b21c0ecc367b6883693bcbf53010941dcd5e6)"
         },
         {
            "address" : "139.180.195.214:11625",
            "id" : "GD5MQ",
            "olver" : 9,
            "ver" : "v11.3.0rc1"
         }
      ]
   },
   "pending_peers" : {
      "inbound" : null,
      "outbound" : null
   }
}
  • stellar-core http-command 'quorum'
2019-10-24T08:27:38.275 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:27:38.280 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 505
Content-Type: application/json

2019-10-24T08:27:38.289 GB6R7 [default INFO] {
   "node" : "GD65K",
   "qset" : {
      "agree" : 3,
      "delayed" : null,
      "disagree" : null,
      "fail_at" : 2,
      "fail_with" : [ "sdf2", "sdf3" ],
      "hash" : "59d361",
      "ledger" : 1341481,
      "missing" : null,
      "phase" : "EXTERNALIZE",
      "value" : {
         "t" : 2,
         "v" : [ "sdf2", "sdf3", "sdf1" ]
      }
   },
   "transitive" : {
      "critical" : null,
      "intersection" : true,
      "last_check_ledger" : 1329406,
      "node_count" : 4
   }
}
  • stellar-core http-command 'quorum?transitive=true'
2019-10-24T08:30:16.574 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:30:16.575 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1403
Content-Type: application/json

2019-10-24T08:30:16.579 GCCHI [default INFO] {
   "critical" : null,
   "intersection" : true,
   "last_check_ledger" : 1329406,
   "node_count" : 4,
   "nodes" : [
      {
         "distance" : 0,
         "heard" : 1341508,
         "node" : "GD65K",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "tracking",
         "value" : "[ txH: 094e3b, ct: 1571905814, upgrades: [ ] ]",
         "value_id" : 1
      },
      {
         "distance" : 1,
         "heard" : 1341508,
         "node" : "sdf2",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "tracking",
         "value" : "[ txH: 094e3b, ct: 1571905814, upgrades: [ ] ]",
         "value_id" : 1
      },
      {
         "distance" : 1,
         "heard" : 1341508,
         "node" : "sdf3",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "tracking",
         "value" : "[ txH: 094e3b, ct: 1571905814, upgrades: [ ] ]",
         "value_id" : 1
      },
      {
         "distance" : 1,
         "heard" : 1341508,
         "node" : "sdf1",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "tracking",
         "value" : "[ txH: 094e3b, ct: 1571905814, upgrades: [ ] ]",
         "value_id" : 1
      }
   ]
}

Instance 2 data

  • cat /opt/stellar/core/etc/stellar-core.cfg 
HTTP_PORT=11626
PUBLIC_HTTP_PORT=true
LOG_FILE_PATH=""

NETWORK_PASSPHRASE="Test SDF Network ; September 2015"

KNOWN_PEERS=[
"core-testnet1.stellar.org",
"core-testnet2.stellar.org",
"core-testnet3.stellar.org"]

DATABASE="postgresql://dbname=core host=XXX user=XXX password=XXX"
UNSAFE_QUORUM=true
FAILURE_SAFETY=1
CATCHUP_RECENT=8640

#The public keys of the Stellar testnet servers
[QUORUM_SET]
THRESHOLD_PERCENT=51 # rounded up -> 2 nodes out of 3
VALIDATORS=[
"GDKXE2OZMJIPOSLNA6N6F2BVCI3O777I2OOC4BV7VOYUEHYX7RTRYA7Y  sdf1",
"GCUCJTIYXSOXKBSNFGNFWW5MUQ54HKRPGJUTQFJ5RQXZXNOLNXYDHRAP  sdf2",
"GC2V2EFSXN6SQTWVYA5EPJPBWWIMSD2XQNKUOHGEKB535AQE2I6IXV2Z  sdf3"]

[HISTORY.cache]
get="cp /opt/stellar/history-cache/{0} {1}"

#The history store of the Stellar testnet
[HISTORY.h1]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_001/{0} -o {1}"

[HISTORY.h2]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_002/{0} -o {1}"

[HISTORY.h3]
get="curl -sf http://history.stellar.org/prd/core-testnet/core_testnet_003/{0} -o {1}"
  • stellar-core http-command 'info'
stellar-core http-command 'info'
2019-10-24T08:13:46.829 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:13:46.831 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1207
Content-Type: application/json

2019-10-24T08:13:46.835 GCPYF [default INFO] {
   "info" : {
      "build" : "v11.3.0rc1",
      "history_failure_rate" : "0.25",
      "ledger" : {
         "age" : 836023,
         "baseFee" : 100,
         "baseReserve" : 5000000,
         "closeTime" : 1571068803,
         "hash" : "e70b405e62cf125668422aa7a59bfcfc2910cfbba7d1eda9e1755567ba95712d",
         "maxTxSetSize" : 100,
         "num" : 1187897,
         "version" : 11
      },
      "network" : "Test SDF Network ; September 2015",
      "peers" : {
         "authenticated_count" : 8,
         "pending_count" : 0
      },
      "protocol_version" : 11,
      "quorum" : {
         "node" : "GBDSA",
         "qset" : {
            "agree" : 3,
            "delayed" : 0,
            "disagree" : 0,
            "fail_at" : 2,
            "hash" : "59d361",
            "ledger" : 1341329,
            "missing" : 0,
            "phase" : "EXTERNALIZE"
         },
         "transitive" : {
            "critical" : null,
            "intersection" : true,
            "last_check_ledger" : 327531,
            "node_count" : 4
         }
      },
      "startedOn" : "2019-08-21T07:45:57Z",
      "state" : "Catching up",
      "status" : [ "Catching up: Running: catchup" ]
   }
}
  • stellar-core http-command 'peers'
2019-10-24T08:23:51.683 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:23:51.685 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1730
Content-Type: application/json

2019-10-24T08:23:51.688 GCOON [default INFO] {
   "authenticated_peers" : {
      "inbound" : null,
      "outbound" : [
         {
            "address" : "13.114.104.43:11625",
            "id" : "GAEPM",
            "olver" : 9,
            "ver" : "stellar-core 11.1.0 (324c1bd61b0e9bada63e0d696d799421b00a7950)"
         },
         {
            "address" : "18.140.91.137:11625",
            "id" : "GBAVU",
            "olver" : 9,
            "ver" : "stellar-core 11.1.0 (324c1bd61b0e9bada63e0d696d799421b00a7950)"
         },
         {
            "address" : "157.245.72.119:11625",
            "id" : "GBDU7",
            "olver" : 9,
            "ver" : "stellar-core 11.4.0 (38f51d60bd7aa473edea6038aaf13c255d79c14d)"
         },
         {
            "address" : "3.87.170.96:11625",
            "id" : "GCCM5",
            "olver" : 10,
            "ver" : "stellar-core 12.0.0 (de0b21c0ecc367b6883693bcbf53010941dcd5e6)"
         },
         {
            "address" : "3.218.204.21:11625",
            "id" : "GCUWV",
            "olver" : 10,
            "ver" : "stellar-core 12.0.0 (de0b21c0ecc367b6883693bcbf53010941dcd5e6)"
         },
         {
            "address" : "54.169.179.214:11625",
            "id" : "GDJ5P",
            "olver" : 10,
            "ver" : "v12.0.0"
         },
         {
            "address" : "152.32.170.134:11625",
            "id" : "GDXSR",
            "olver" : 10,
            "ver" : "stellar-core 12.0.0 (de0b21c0ecc367b6883693bcbf53010941dcd5e6)"
         },
         {
            "address" : "139.180.195.214:11625",
            "id" : "GD5MQ",
            "olver" : 9,
            "ver" : "v11.3.0rc1"
         }
      ]
   },
   "pending_peers" : {
      "inbound" : null,
      "outbound" : null
   }
}
  • stellar-core http-command 'quorum'
2019-10-24T08:26:36.888 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:26:36.892 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 504
Content-Type: application/json

2019-10-24T08:26:36.898 GBICN [default INFO] {
   "node" : "GBDSA",
   "qset" : {
      "agree" : 3,
      "delayed" : null,
      "disagree" : null,
      "fail_at" : 2,
      "fail_with" : [ "sdf2", "sdf3" ],
      "hash" : "59d361",
      "ledger" : 1341469,
      "missing" : null,
      "phase" : "EXTERNALIZE",
      "value" : {
         "t" : 2,
         "v" : [ "sdf2", "sdf3", "sdf1" ]
      }
   },
   "transitive" : {
      "critical" : null,
      "intersection" : true,
      "last_check_ledger" : 327531,
      "node_count" : 4
   }
}
  • stellar-core http-command 'quorum?transitive=true'
2019-10-24T08:29:17.089 <startup> [default INFO] Config from stellar-core.cfg
2019-10-24T08:29:17.091 <startup> [default INFO] Using QUORUM_SET: {
   "t" : 2,
   "v" : [ "sdf1", "sdf2", "sdf3" ]
}

Content-Length: 1429
Content-Type: application/json

2019-10-24T08:29:17.095 GAR3A [default INFO] {
   "critical" : null,
   "intersection" : true,
   "last_check_ledger" : 327531,
   "node_count" : 4,
   "nodes" : [
      {
         "distance" : 0,
         "heard" : 1341497,
         "node" : "GBDSA",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "tracking",
         "value" : "[ txH: 17b978, ct: 1571905748, upgrades: [ ] ]",
         "value_id" : 1
      },
      {
         "distance" : 1,
         "heard" : 1341498,
         "node" : "sdf2",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "ahead",
         "value" : "[ SIGNED@sdf2 txH: 865dee, ct: 1571905756, upgrades: [ ] ]",
         "value_id" : 2
      },
      {
         "distance" : 1,
         "heard" : 1341498,
         "node" : "sdf3",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "ahead",
         "value" : "[ SIGNED@sdf3 txH: 865dee, ct: 1571905756, upgrades: [ ] ]",
         "value_id" : 3
      },
      {
         "distance" : 1,
         "heard" : 1341498,
         "node" : "sdf1",
         "qset" : {
            "t" : 2,
            "v" : [ "sdf2", "sdf3", "sdf1" ]
         },
         "status" : "ahead",
         "value" : "[ SIGNED@sdf1 txH: 865dee, ct: 1571905756, upgrades: [ ] ]",
         "value_id" : 4
      }
   ]
}
Improve this question

1 Answer 1

Reset to default
0

Looks to me that you were (are still?) using a version of stellar-core not compatible with the network. "build" : "v11.3.0rc1" doesn’t support protocol 12 (network upgraded on Monday).

As of testnet, same thing (you need v12.1.0 or above), but you also need to rebuild it from scratch (newdb) as Testnet was just rebuilt yesterday.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.