Lost ledger data: what is the proper way to shutdown and restart a node?

Question

I had to restart Docker Engine, and I stopped the services by attaching to the container and issuing ctrl+c. It was in the process of downloading ledger files, and I thought that it would pick up from that point after a restart. It didn't. Instead, it now has to start from the beginning.

Is there a graceful way to shutdown core, horizon and postgres? Why does it wipe all ledger history and start from the beginning upon node restart?

Update It looks like the catchup data and history is being stored in a temporary directory!!

root@stellar:/# du -sch /opt/stellar/core/buckets/tmp/
71M     /opt/stellar/core/buckets/tmp/
71M     total
root@stellar:/# du -sch /opt/stellar/core/buckets/tmp/
73M     /opt/stellar/core/buckets/tmp/
73M     total
root@stellar:/# ls -lah /opt/stellar/core/buckets/tmp/
total 144K
drwx------ 4 stellar stellar 4.0K Aug 14 21:34 .
drwx------ 3 stellar stellar 4.0K Aug 14 21:33 ..
drwx------ 2 stellar stellar 132K Aug 14 21:40 catchup-2d8a3980ddf4fc04
drwx------ 2 stellar stellar 4.0K Aug 14 21:36 history-d3c4ab93652f11be

How do I make this permanent?

Launch options:

stellard:
    command:
    - --pubnet
    container_name: stellard
    hostname: stellar.fullnode
    image: docker.dev.ruvpfs.swatt.exchange/stellar-full-node:base
    networks:
      fullnodes:
        ipv4_address: 172.20.0.13
    ports:
    - 10.99.125.151:8000:8000/tcp
    volumes:
    - /mnt/XLM:/opt/stellar:rw

Output from stellar-core after restart:

root@stellar:/# stellar-core --c info
Content-Length: 904
Content-Type: application/json

2018-08-14T21:15:22.988 GCQZE [default INFO] {
   "info" : {
      "build" : "v9.2.0",
      "ledger" : {
         "age" : 1534281322,
         "baseFee" : 100,
         "baseReserve" : 100000000,
         "closeTime" : 0,
         "hash" : "39c2a3cd4141b2853e70d84601faa44744660334b48f3228e0309342e3f4eb48",
         "num" : 2,
         "version" : 0
      },
      "network" : "Public Global Stellar Network ; September 2015",
      "peers" : {
         "authenticated_count" : 8,
         "pending_count" : 0
      },
      "protocol_version" : 9,
      "quorum" : {
         "19460876" : {
            "agree" : 5,
            "disagree" : 0,
            "fail_at" : 2,
            "hash" : "ba2fc8",
            "missing" : 0,
            "phase" : "EXTERNALIZE"
         }
      },
      "startedOn" : "2018-08-14T20:55:23Z",
      "state" : "Catching up",
      "status" : [ "Catching up: downloading ledger files 10114/304073 (3%)" ]
   }
}

My config:

    HTTP_PORT=11626
    PUBLIC_HTTP_PORT=true
    LOG_FILE_PATH=""

    DATABASE="postgresql://dbname=core host=localhost user=stellar password=xxxx"
    NETWORK_PASSPHRASE="Public Global Stellar Network ; September 2015"
    CATCHUP_COMPLETE=true
    # MAX_CONCURRENT_SUBPROCESSES (integer) default 16
    # History catchup can potentialy spawn a bunch of sub-processes.
    # This limits the number that will be active at a time.
    MAX_CONCURRENT_SUBPROCESSES=24

    # TARGET_PEER_CONNECTIONS (Integer) default 8
    # This controls how aggressively the server will connect to other peers.
    # It will send outbound connection attempts until it is at this
    #   number of peer connections.
    TARGET_PEER_CONNECTIONS=8

    NODE_NAMES=[
    "GAOO3LWBC4XF6VWRP5ESJ6IBHAISVJMSBTALHOQM2EZG7Q477UWA6L7U  eno",
    "GAXP5DW4CVCW2BJNPFGTWCEGZTJKTNWFQQBE5SCWNJIJ54BOHR3WQC3W  moni",
    "GBFZFQRGOPQC5OEAWO76NOY6LBRLUNH4I5QYPUYAK53QSQWVTQ2D4FT5  dzham",
    "GDXWQCSKVYAJSUGR2HBYVFVR7NA7YWYSYK3XYKKFO553OQGOHAUP2PX2  jianing",
    "GCJCSMSPIWKKPR7WEPIQG63PDF7JGGEENRC33OKVBSPUDIRL6ZZ5M7OO  tempo.eu.com",
    "GCCW4H2DKAC7YYW62H3ZBDRRE5KXRLYLI4T5QOSO6EAMUOE37ICSKKRJ  sparrow_tw",
    "GD5DJQDDBKGAYNEAXU562HYGOOSYAEOO6AS53PZXBOZGCP5M2OPGMZV3  fuxi.lab",
    "GBGGNBZVYNMVLCWNQRO7ASU6XX2MRPITAGLASRWOWLB4ZIIPHMGNMC4I  huang.lab",
    "GDPJ4DPPFEIP2YTSQNOKT7NMLPKU2FFVOEIJMG36RCMBWBUR4GTXLL57  nezha.lab",
    "GCDLFPQ76D6YUSCUECLKI3AFEVXFWVRY2RZH2YQNYII35FDECWUGV24T  SnT.Lux",
    "GBAR4OY6T6M4P344IF5II5DNWHVUJU7OLQPSMG2FWVJAFF642BX5E3GB  telindus",
    # non validating
    "GCGB2S2KGYARPVIA37HYZXVRM2YZUEXA6S33ZU5BUDC6THSB62LZSTYH  sdf_watcher1",
    "GCM6QMP3DLRPTAZW2UZPCPX2LF3SXWXKPMP3GKFZBDSF3QZGV2G5QSTK  sdf_watcher2",
    "GABMKJM6I25XI4K7U6XWMULOUQIQ27BCTMLS6BYYSOWKTBUXVRJSXHYQ  sdf_watcher3",
    # seem down
    "GB6REF5GOGGSEHZ3L2YK6K4T4KX3YDMWHDCPMV7MZJDLHBDNZXEPRBGM  donovan",
    "GBGR22MRCIVW2UZHFXMY5UIBJGPYABPQXQ5GGMNCSUM2KHE3N6CNH6G5  nelisky1",
    "GA2DE5AQF32LU5OZ5OKAFGPA2DLW4H6JHPGYJUVTNS3W7N2YZCTQFFV6  nelisky2",
    "GDJ73EX25GGUVMUBCK6DPSTJLYP3IC7I3H2URLXJQ5YP56BW756OUHIG  w00kie",
    "GAM7A32QZF5PJASRSGVFPAB36WWTHCBHO5CHG3WUFTUQPT7NZX3ONJU4  ptarasov"
    ]

    # NODE_IS_VALIDATOR (boolean) default false.
    # Only nodes that want to participate in SCP should set NODE_IS_VALIDATOR=true.
    # Most instances should operate in observer mode with NODE_IS_VALIDATOR=false.
    # See QUORUM_SET below.
    NODE_IS_VALIDATOR=false

    KNOWN_PEERS=[
    "core-live-a.stellar.org:11625",
    "core-live-b.stellar.org:11625",
    "core-live-c.stellar.org:11625",
    "confucius.strllar.org",
    "stellar1.bitventure.co",
    "stellar.256kw.com"]



 [QUORUM_SET]
    VALIDATORS=[
    "$sdf_watcher1","$eno","$tempo.eu.com","$sdf_watcher2","$sdf_watcher3"
   [HISTORY.cache]
get="cp /opt/stellar/history-cache/{0} {1}"

# Stellar.org history store
[HISTORY.stellar]
get="curl http://history.stellar.org/{0} -o {1}"
# Stellar.org history store
[HISTORY.sdf1]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_001/{0} -o {1}"

[HISTORY.sdf2]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_002/{0} -o {1}"

[HISTORY.sdf3]
get="curl -sf http://history.stellar.org/prd/core-live/core_live_003/{0} -o {1}"

Answer 1

It's better to stop the container using standard docker stop command, like docker stop stellar_quickstart.

Make sure that your docker container is running in persistent mode, otherwise you will lose current state (the database and all history) when the container is stopped.

Answer 2

I think I have a better idea of how the stack works, and when it's safe to bring down the node without losing data.

If you're running CATCHUP_COMPLETE=true like me, core will first download the ledger files, then transaction files and then move on to the next checkpoint. It looks something like this:

"state" : "Catching up",
      "status" : [ "Catching up: downloading ledger files 184888/304080 (60%)" ]

"state" : "Catching up",
      "status" : [ "Catching up: downloading transactions files 297092/304080 (97%)" ]

 "state" : "Catching up",
      "status" : [ "Catching up: applying checkpoint 21348/304080 (7%)" ]
   }

The entire time it's downloading ledger files, those go into the /opt/stellar/core/buckets/tmp/ directory, meaning if you stop/start, you will lose that progress. You can verify what ledgers you actually have on disk by going into the postgres db and running a count query: (thanks @Orbit Lens)

core=> select count(*) from ledgerheaders;
 count
-------
     1
(1 row)

Once the transaction files catchup process completed for the first checkpoint, I noticed that my ledgerheaders finally started increasing (persistent data):

core=> select count(*) from ledgerheaders;
  count
---------
 1517083
(1 row)

Now I'm pretty confident that if my docker container goes down for whatever reason, it will at least pick up from the 2nd checkpoint.