5

I have a private network of Stellar core nodes running on docker swarm with the DB and ledger data persisted correctly. When I remove or update the service for maintenance/upgrade of the docker image and then restart it I often see this:

2019-03-04T04:43:12.761 GAWSB [default INFO] Loading last known ledger
2019-03-04T04:43:12.763 GAWSB [Ledger WARNING] Some buckets are missing in 'buckets'.
2019-03-04T04:43:12.763 GAWSB [Ledger WARNING] Attempting to recover from the history store.
2019-03-04T04:43:12.763 GAWSB [History INFO] Starting RepairMissingBucketsWork
2019-03-04T04:43:12.818 GAWSB [Process WARNING] process 118 exited 1: cp /tmp/stellar-core/history/vs/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz buckets/tmp/repair-buckets-7460063929a1b56d/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz.tmp
2019-03-04T04:43:12.819 GAWSB [Work WARNING] Reached retry limit 0 for get-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz
2019-03-04T04:43:12.819 GAWSB [Work WARNING] Scheduling retry #1/32 in 1 sec, for get-and-unzip-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz
2019-03-04T04:43:13.829 GAWSB [Process WARNING] process 122 exited 1: cp /tmp/stellar-core/history/vs/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz buckets/tmp/repair-buckets-7460063929a1b56d/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz.tmp
2019-03-04T04:43:13.829 GAWSB [Work WARNING] Reached retry limit 0 for get-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz

These buckets don't exist and I presume it is because the container was killed in the middle of a bucket write. There are issues here and here which seem to suggest the only fix for this is to run newdb which is not going to work in production...

Is there a clean way of stopping Stellar Core on a container shutdown so that this issue is avoided?

Improve this question
4
  • I would actually like to know how this might affect stellar core when installed directly on an OS as opposed to Docker as well. Are there some best practices available on updating or performing network maintenance without destroying your ledger?
    – Owen
    Mar 4, 2019 at 7:09
  • I should add: I am using the latest version of Stellar Core: 10.2.0-810-54504c71
    – Owen
    Mar 4, 2019 at 7:20
  • did you try with stellar-core --forcescp after restart?
    – cesarm
    Mar 4, 2019 at 7:59
  • Yes the docker entry point always runs force scp before starting Stellar
    – Owen
    Mar 4, 2019 at 9:33

1 Answer 1

Reset to default
2

Experience now allows me to update this answer for googlers of the future. When running Stellar Core in Docker you must persist the following:

  1. Database state

  2. Set BUCKET_DIR_PATH in your stellar-core.cfg and persist this location outside of the container

  3. If you put to a local history archive also ensure this location is persisted outside of the container

Improve this answer
2
  • Why must BUCKET_DIR_PATH location and local history archive be persisted outside of the container? If I have the container mounted to the directory on the server and these directories located inside that what problems occur? Thanks in advance, and thanks for adding the answer to your question.
    – John Murphy
    Jun 18, 2019 at 14:02
  • 1
    As far as I know there are basically two steps to long term storage of the ledger in Stellar. First the recent state (I believe it is the last 5 minutes) data is stored in the DB and also stored in the BUCKET_DIR_PATH. After 5 minutes it is then published to the history archive. If you don't persist these outside the container and take the container down you have a hole in your nodes history that requires a catchup - this was an issue in my original question because it turned out all my nodes were missing this piece of history and so the ledger was corrupted.
    – Owen
    Jun 20, 2019 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.