3

I have a private network of Stellar core nodes running on docker swarm with the DB and ledger data persisted correctly. When I remove or update the service for maintenance/upgrade of the docker image and then restart it I often see this:

2019-03-04T04:43:12.761 GAWSB [default INFO] Loading last known ledger
2019-03-04T04:43:12.763 GAWSB [Ledger WARNING] Some buckets are missing in 'buckets'.
2019-03-04T04:43:12.763 GAWSB [Ledger WARNING] Attempting to recover from the history store.
2019-03-04T04:43:12.763 GAWSB [History INFO] Starting RepairMissingBucketsWork
2019-03-04T04:43:12.818 GAWSB [Process WARNING] process 118 exited 1: cp /tmp/stellar-core/history/vs/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz buckets/tmp/repair-buckets-7460063929a1b56d/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz.tmp
2019-03-04T04:43:12.819 GAWSB [Work WARNING] Reached retry limit 0 for get-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz
2019-03-04T04:43:12.819 GAWSB [Work WARNING] Scheduling retry #1/32 in 1 sec, for get-and-unzip-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz
2019-03-04T04:43:13.829 GAWSB [Process WARNING] process 122 exited 1: cp /tmp/stellar-core/history/vs/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz buckets/tmp/repair-buckets-7460063929a1b56d/bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz.tmp
2019-03-04T04:43:13.829 GAWSB [Work WARNING] Reached retry limit 0 for get-remote-file bucket/61/83/ea/bucket-6183ea169e9b74480c793417fa42694cc961df5e47d4ca9e9a89164ceadccfa8.xdr.gz

These buckets don't exist and I presume it is because the container was killed in the middle of a bucket write. There are issues here and here which seem to suggest the only fix for this is to run newdb which is not going to work in production...

Is there a clean way of stopping Stellar Core on a container shutdown so that this issue is avoided?

  • I would actually like to know how this might affect stellar core when installed directly on an OS as opposed to Docker as well. Are there some best practices available on updating or performing network maintenance without destroying your ledger? – Owen Mar 4 at 7:09
  • I should add: I am using the latest version of Stellar Core: 10.2.0-810-54504c71 – Owen Mar 4 at 7:20
  • did you try with stellar-core --forcescp after restart? – cesarm Mar 4 at 7:59
  • Yes the docker entry point always runs force scp before starting Stellar – Owen Mar 4 at 9:33
1

Experience now allows me to update this answer for googlers of the future. When running Stellar Core in Docker you must persist the following:

  1. Database state

  2. Set BUCKET_DIR_PATH in your stellar-core.cfg and persist this location outside of the container

  3. If you put to a local history archive also ensure this location is persisted outside of the container

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.