Publishing history to new history archive results in error of loading history state

Question

I keep getting the [History ERROR] Error loading history state: rapidjson internal assertion failure: IsObject() when trying to publish the state of a recently added node to it's recently initialized history archive on S3, related log output below:

...
2019-05-01T10:30:25.051 GC2R7 [Work DEBUG] scheduling completion of get-history-archive-state
2019-05-01T10:30:25.051 GC2R7 [Work DEBUG] completed get-history-archive-state
2019-05-01T10:30:25.051 GC2R7 [Work DEBUG] work get-history-archive-state : WORK_RUNNING -> WORK_FAILURE_RETRY
2019-05-01T10:30:25.051 GC2R7 [Work WARNING] Scheduling retry #5/5 in 28 sec, for get-history-archive-state
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] resetting get-history-archive-state
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] work get-history-archive-state : WORK_FAILURE_RETRY -> WORK_PENDING
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] resetting get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] advancing get-history-archive-state
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] advancing get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] all 0 children of get-remote-file .well-known/stellar-history.json successful, scheduling run
2019-05-01T10:30:53.051 GC2R7 [Work DEBUG] scheduling run of get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.052 GC2R7 [Work DEBUG] starting get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.052 GC2R7 [Process DEBUG] Running: curl -sf https://<my-bucket>.s3.amazonaws.com/.well-known/stellar-history.json -o /home/ubuntu/buckets/tmp/history-30a480ac3f3357b4/my-bucket-stellar-history.json
2019-05-01T10:30:53.052 GC2R7 [Work DEBUG] running get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.052 GC2R7 [Work DEBUG] work get-remote-file .well-known/stellar-history.json : WORK_PENDING -> WORK_RUNNING
2019-05-01T10:30:53.382 GC2R7 [Process DEBUG] found 1 child processes that terminated
2019-05-01T10:30:53.382 GC2R7 [Process DEBUG] process 2907 exited 0: curl -sf https://<my-bucket>.s3.amazonaws.com/.well-known/stellar-history.json -o /home/ubuntu/buckets/tmp/history-30a480ac3f3357b4/my-bucket-stellar-history.json
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] completed get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] work get-remote-file .well-known/stellar-history.json : WORK_RUNNING -> WORK_SUCCESS
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] notifying parent of successful get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] notified get-history-archive-state of completed child get-remote-file .well-known/stellar-history.json
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] advancing get-history-archive-state
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] all 1 children of get-history-archive-state successful, scheduling run
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] scheduling run of get-history-archive-state
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] starting get-history-archive-state
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] running get-history-archive-state
2019-05-01T10:30:53.382 GC2R7 [Work DEBUG] work get-history-archive-state : WORK_PENDING -> WORK_RUNNING
2019-05-01T10:30:53.382 GC2R7 [History ERROR] Error loading history state: rapidjson internal assertion failure: IsObject() [GetHistoryArchiveStateWork.cpp:73]
2019-05-01T10:30:53.383 GC2R7 [History ERROR] It may be problem with local filesystem. Ensure that there is enough space to perform that operation and that disc is behaving correctly. [GetHistoryArchiveStateWork.cpp:74]
2019-05-01T10:30:53.383 GC2R7 [History ERROR] OR [GetHistoryArchiveStateWork.cpp:75]
2019-05-01T10:30:53.383 GC2R7 [History ERROR] One or more of history archives may be corrupted. Update HISTORY configuration entry to only contain valid ones [GetHistoryArchiveStateWork.cpp:76]
2019-05-01T10:30:53.383 GC2R7 [History ERROR] OR [GetHistoryArchiveStateWork.cpp:77]
2019-05-01T10:30:53.383 GC2R7 [History ERROR] Upgrade this stellar-core installation to newest version [GetHistoryArchiveStateWork.cpp:78]
...

I currently have a working private stellar network with 3 nodes. I wanted to add a new node to the network, so I did the following:

• set up a new instance with the same config as the other instances and a new history archive called my-bucket to write to and read from, as well as referencing the other nodes' history archives to read from

• run new-hist my-bucket and new-db on the new instance

• run catchup current/max to fetch the history from the other nodes' history archive. This runs for a while and ends without errors.

• I noticed that the new history archive is at this point still in it's initial state (no checkpoints published), so I run the publish command and I keep getting the log output above, though it's gradually publishing the ledgers and they appear in the new history bucket.

But I'm concerned about that internal assertion failure while loading the history state. Will it disappear when the new history bucket is completely published?

below is my stellar config for reference:

HTTP_PORT=11626
PUBLIC_HTTP_PORT=true

LOG_FILE_PATH=""

COMMANDS=["ll?level=debug"]

DATABASE="postgresql://dbname=stellar_test host=localhost port=5432 user=stellar password=<xxx>"
BUCKET_DIR_PATH="/home/ubuntu/buckets"

NETWORK_PASSPHRASE="cccccc"

TARGET_PEER_CONNECTIONS=3

KNOWN_PEERS=[
 "<ip1>",
 "<ip2>",
 "<ip3>"

]

CATCHUP_RECENT=8192

NODE_SEED="<some seed>"

NODE_IS_VALIDATOR=true

NODE_NAMES=[
 "<addr4> stellar-node-4",
 "<addr3>  stellar-node-1",
 "<addr2>  stellar-node-2",
 "<addr1>  stellar-node-3"

]

FAILURE_SAFETY=0
UNSAFE_QUORUM=true

[QUORUM_SET]
THRESHOLD_PERCENT=66
VALIDATORS=[
 "$stellar-node-4",

 "$stellar-node-1",
 "$stellar-node-2",
 "$stellar-node-3"

]


[HISTORY.my-bucket]
get="curl -sf https://<my-bucket>.s3.amazonaws.com/{0} -o {1}"
put="aws s3 cp {0} s3://<my-bucket>/{1} --no-sign-request"

[HISTORY.stellar-node-1]
get="curl -sf https://<bucket1>.s3.amazonaws.com/{0} -o {1}"

#[HISTORY.stellar-node-2]
#get="curl -sf https://<bucket2>.s3.amazonaws.com/{0} -o {1}"

#[HISTORY.stellar-node-3]
#get="curl -sf <bucket3>.s3.amazonaws.com/{0} -o {1}"

The nodes use stellar-core 11.0.0 (236f831521b6724c0ae63906416faa997ef27e19)

Answer 1

Hmmmm, typically this error means that rapidjson, the json parser, is saying it can't digest a history archive state. Usually this happens when a history archive is misconfigured and you've made the "get" command be curl ... alone rather than curl -sf ...

However, your commands in your configuration look correct to me, so I would likely try to verify that you're actually pulling over the wire what you'd expect.