<< Background >>
Private network of 4 nodes on 4 VMs, hosted on same server.
Core and Horizon built on 1 VM, and then 3 copies were made from this VM. NODE_SEED (core) & corresponding IP (Horizon & friendbot) were changed correspondingly.
All Horizons have the flag "--ingest=true", and other flags identical except for friendbot IP.
As I submitted a transaction to node1, the reponse looks like:
{ _links:
{ transaction:
{ href: 'http://192.168.7.81:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec' } },
hash: 'e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec',
ledger: 123501,
envelope_xdr: 'AAAAAOtyZp2+5xgJ5ugpgr2IJn72iZzB4dkFLP34zSKLAvn4AAAAZAAAAUQAACGLAAAAAAAAAAAAAAABAAAAAQAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAAAACJVEAAAAAAAAAAAosC+fgAAABAc8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDn1xHN4AAABAtIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ==',
result_xdr: 'AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAA=',
result_meta_xdr: 'AAAAAAAAAAEAAAACAAAAAwAB4jkAAAABAAAAANFBASwV/Roz9YjORuPI2zbluitVXXdORMQr0eH0VTZ1AAAAAkFzczF0bzAAAAAAAAAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAEXjIWKPAf/////////8AAAABAAAAAAAAAAAAAAABAAHibQAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAReMjh+AB//////////wAAAAEAAAAAAAAAAA==' }
Then I curl the href on Horizon of node1 (Res404):
curl http://192.168.7.81:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec
{
"type": "https://stellar.org/horizon-errors/not_found",
"title": "Resource Missing",
"status": 404,
"detail": "The resource at the url requested was not found. This is usually occurs for one of two reasons: The url requested is not valid, or no data in our database could be found with the parameters provided."
}
While curl node2 gives the correct info (Res200):
curl http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec
{
"_links": {
"self": {
"href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec"
},
"account": {
"href": "http://192.168.7.82:8000/accounts/GDVXEZU5X3TRQCPG5AUYFPMIEZ7PNCM4YHQ5SBJM7X4M2IULAL47R45Q"
},
"ledger": {
"href": "http://192.168.7.82:8000/ledgers/123501"
},
"operations": {
"href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec/operations{?cursor,limit,order}",
"templated": true
},
"effects": {
"href": "http://192.168.7.82:8000/transactions/e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec/effects{?cursor,limit,order}",
"templated": true
},
"precedes": {
"href": "http://192.168.7.82:8000/transactions?order=asc\u0026cursor=530432756027392"
},
"succeeds": {
"href": "http://192.168.7.82:8000/transactions?order=desc\u0026cursor=530432756027392"
}
},
"id": "e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec",
"paging_token": "530432756027392",
"hash": "e148053e6eb998ab3b8c8eb2e316054a58a602223f1e7978ce2cc9c08b5230ec",
"ledger": 123501,
"created_at": "2018-03-13T06:17:32Z",
"source_account": "GDVXEZU5X3TRQCPG5AUYFPMIEZ7PNCM4YHQ5SBJM7X4M2IULAL47R45Q",
"source_account_sequence": "1391569412491",
"fee_paid": 100,
"operation_count": 1,
"envelope_xdr": "AAAAAOtyZp2+5xgJ5ugpgr2IJn72iZzB4dkFLP34zSKLAvn4AAAAZAAAAUQAACGLAAAAAAAAAAAAAAABAAAAAQAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAAAACJVEAAAAAAAAAAAosC+fgAAABAc8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDn1xHN4AAABAtIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ==",
"result_xdr": "AAAAAAAAAGQAAAAAAAAAAQAAAAAAAAABAAAAAAAAAAA=",
"result_meta_xdr": "AAAAAAAAAAEAAAACAAAAAwAB4jkAAAABAAAAANFBASwV/Roz9YjORuPI2zbluitVXXdORMQr0eH0VTZ1AAAAAkFzczF0bzAAAAAAAAAAAACOFIVTgnqFCLqO/8a9EsD/3PRh/+50+TZRJAlefXEc3gAAEXjIWKPAf/////////8AAAABAAAAAAAAAAAAAAABAAHibQAAAAEAAAAA0UEBLBX9GjP1iM5G48jbNuW6K1Vdd05ExCvR4fRVNnUAAAACQXNzMXRvMAAAAAAAAAAAAI4UhVOCeoUIuo7/xr0SwP/c9GH/7nT5NlEkCV59cRzeAAAReMjh+AB//////////wAAAAEAAAAAAAAAAA==",
"fee_meta_xdr": "AAAAAgAAAAMAAeI5AAAAAAAAAADrcmadvucYCeboKYK9iCZ+9omcweHZBSz9+M0iiwL5+AAAABdIac4YAAABRAAAIYoAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAEAAeJtAAAAAAAAAADrcmadvucYCeboKYK9iCZ+9omcweHZBSz9+M0iiwL5+AAAABdIac20AAABRAAAIYsAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA==",
"memo_type": "none",
"signatures": [
"c8d0ngvx2Nux+oTSe1qrndaxG10jRgeP+M1N22csIGQW1W6jXA+NKxw1A/VB27uiQls85ih3tS4ndNW/78QRDg==",
"tIXOwrHTYdzQuqyHdjiNZ9ayvvA969xZ7cZdxRiypGcepb7YCcMBom40mZZMqi0uDpqCMV8+f0yJr4FJkL2sBQ=="
]
}
Horizon DB on all 4 nodes checked, and only node2 has the new record; node1,3,4 has no such record.
Looking into the Horizon logs of node1,3,4, this message repeats:
"import session failed: failed to load header: sql: no rows in result set" pid=8077
Any suggestions for the cause/solution?
===================
p.s. Querying the involved account gives the same asset amounts on all 4 nodes. So this seems not to be an issue with core.
