1

<< Background >>

Private network of 4 nodes on 4 VMs, hosted on same server.

Core and Horizon built on 1 VM, and then 3 copies were made from this VM. NODE_SEED (core) & corresponding IP (Horizon & friendbot) were changed correspondingly.

All Horizons have the flag "--ingest=true", and other flags identical except for friendbot IP.

As I submitted a transaction to node1, the reponse looks like:

{ _links:
   { transaction:
      { href: 'http://192.168.7.81:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec' } },
  hash: 'e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec',
  ledger: 123501,
  envelope_xdr: 'AAAAAOtyZp2+5xgJ5ugpgr2IJn72iZzB4dkFLP34zSKLAvn4AAAAZAAAAUQAACGLAAAAAAAAAAAAAAABAAAAAQAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAAAACJVEAAAAAAAAAAAosC+fgAAABAc8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDn1xHN4AAABAtIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ==',
  result_xdr: 'AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAA=',
  result_meta_xdr: 'AAAAAAAAAAEAAAACAAAAAwAB4jkAAAABAAAAANFBASwV/Roz9YjORuPI2zbluitVXXdORMQr0eH0VTZ1AAAAAkFzczF0bzAAAAAAAAAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAEXjIWKPAf/////////8AAAABAAAAAAAAAAAAAAABAAHibQAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAReMjh+AB//////////wAAAAEAAAAAAAAAAA==' }

Then I curl the href on Horizon of node1 (Res404):

curl http://192.168.7.81:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec

{
  "type": "https://stellar.org/horizon-errors/not_found",
  "title": "Resource Missing",
  "status": 404,
  "detail": "The resource at the url requested was not found.  This is usually occurs for one of two reasons:  The url requested is not valid, or no data in our database could be found with the parameters provided."
}

While curl node2 gives the correct info (Res200):

curl http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec

{
  "_links": {
    "self": {
      "href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec"
    },
    "account": {
      "href": "http://192.168.7.82:8000/accounts/GDVXEZU5X3TRQCPG5AUYFPMIEZ7PNCM4YHQ5SBJM7X4M2IULAL47R45Q"
    },
    "ledger": {
      "href": "http://192.168.7.82:8000/ledgers/123501"
    },
    "operations": {
      "href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec/operations{?cursor,limit,order}",
      "templated": true
    },
    "effects": {
      "href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec/effects{?cursor,limit,order}",
      "templated": true
    },
    "precedes": {
      "href": "http://192.168.7.82:8000/transactions?order=asc\u0026cursor=530432756027392"
    },
    "succeeds": {
      "href": "http://192.168.7.82:8000/transactions?order=desc\u0026cursor=530432756027392"
    }
  },
  "id": "e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec",
  "paging_token": "530432756027392",
  "hash": "e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec",
  "ledger": 123501,
  "created_at": "2018-03-13T06:17:32Z",
  "source_account": "GDVXEZU5X3TRQCPG5AUYFPMIEZ7PNCM4YHQ5SBJM7X4M2IULAL47R45Q",
  "source_account_sequence": "1391569412491",
  "fee_paid": 100,
  "operation_count": 1,
  "envelope_xdr": "AAAAAOtyZp2+5xgJ5ugpgr2IJn72iZzB4dkFLP34zSKLAvn4AAAAZAAAAUQAACGLAAAAAAAAAAAAAAABAAAAAQAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAAAACJVEAAAAAAAAAAAosC+fgAAABAc8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDn1xHN4AAABAtIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ==",
  "result_xdr": "AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAA=",
  "result_meta_xdr": "AAAAAAAAAAEAAAACAAAAAwAB4jkAAAABAAAAANFBASwV/Roz9YjORuPI2zbluitVXXdORMQr0eH0VTZ1AAAAAkFzczF0bzAAAAAAAAAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAEXjIWKPAf/////////8AAAABAAAAAAAAAAAAAAABAAHibQAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAReMjh+AB//////////wAAAAEAAAAAAAAAAA==",
  "fee_meta_xdr": "AAAAAgAAAAMAAeI5AAAAAAAAAADrcmadvucYCeboKYK9iCZ+9omcweHZBSz9+M0iiwL5+AAAABdIac4YAAABRAAAIYoAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAEAAeJtAAAAAAAAAADrcmadvucYCeboKYK9iCZ+9omcweHZBSz9+M0iiwL5+AAAABdIac20AAABRAAAIYsAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA==",
  "memo_type": "none",
  "signatures": [
    "c8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDg==",
    "tIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ=="
  ]
}

Horizon DB on all 4 nodes checked, and only node2 has the new record; node1,3,4 has no such record.

Looking into the Horizon logs of node1,3,4, this message repeats:

"import session failed: failed to load header: sql: no rows in result set" pid=8077

Any suggestions for the cause/solution?

===================

p.s. Querying the involved account gives the same asset amounts on all 4 nodes. So this seems not to be an issue with core.

3

EDIT There'a a new article explaining potential recovery steps: https://www.stellar.org/developers/software/known-issues.html

Looks like you are falling into this issue. stellar-core ledgerheader table can have gaps. For example it has ledgers: 45000-46000 and then the next ledger is ledger: 50000. This can happen when you have CATCHUP_RECENT parameter set and stellar-core was not working for some time.

Currently the only way to fix this is to clear Horizon database.

  • Thanks Bartek! Is there any suggested way to regain the previous Horizon data? This is because I am running a Laboratory on top of Horizon, and the older record matters. (I'll set CATCHUP_COMPLETE for the cores later.) – cesarm Mar 14 '18 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.