I think I had run into the same issue, as Stellar Core by default uses a "fast-sync" mode (instead of "full-sync").
In terms of architecture, Horizon just takes the data stored at Core's DB, i.e. data that Core has processed. **It may be your Core did not process the record before ledger#400000, or at least your Core's DB does not possess the relevant entries. **
By default, Core is doing the
CATCHUP_RECENT mode (i.e. "fast-sync"), as
If a Stellar Core did not start working from ledger#1 (i.e. joining after the network is up, for example at ledger#400005), the Core will catch up at the next checkpoint, but not handle the old old ledgers. It may contain very recent records like ledger#400001 but not #500. As Core did not process #500, Horizon will be unable to take in this record.
Similar problem will occur, when your Core was in sync with the network but is down for a period of time. Using
CATCHUP_RECENT mode will not let you get back the data during your node's downtime.
Configure Core with
CATCHUP_COMPLETE=true, and reset your Core (clear DB, and do things like --newdb & --newhist). Core will then process all the ledgers from #1 to #current, even when it has downtime and did not get certain ledgers.
I would recommend to do the steps node-by-node, so that your previous data is not lost. And it may take long long time in case something went wrong.
From my experience, it took 30min for a new-born node to catchup completely with my 1-week-old network.
After 10 minutes, Core will have entered
"Catching up" state and started to process data, saying
Publishing 166 queued checkpoints [63-10623]. Horizon should be constantly renewing like
"history_latest_ledger": 576, "history_elder_ledger": 31, "core_latest_ledger": 1998550.
After some more time, Core should enter
"Synced!" state before Horizon fully catches up. Horizon is ok done when
Full-sync takes time, just come back later.