So our c5.xlarge instance in AWS is processing about 1k checkpoints per day on average. With somewhere around 275068 checkpoints, it would take around 275 days to complete the full sync. Now the rate of checkpoints processed per day is variable, so that estimation is off, but the process only seems to be slowing as more checkpoints are processed.
Has anyone else experienced this when running a full sync (including writing the archive)?
Last time I started the full sync on my server (dedicated i7, 64 RAM, SSD), it took around 2 days to complete. It was about a month ago, with Stellar Core v9.1.0 and Horizon v0.12.1.
With earlier versions before Stellar Core v9, the full sync process usually took more than 10 days to complete on my environment.
When CATCHUP_COMPLETE=true option set, Stellar Core uses publicly available history archives, which significantly increases the speed of checkpoints application.
As pointed out in the question, catchup can take a lot of time. There are 2 phases to this (at least on the V9 I'm working with): downloading the files from public history buckets and re-playing them. In my experience the downloading phase isnt very efficient on the cores (it tends to fails often and backs off aggressively) and can be sped up by manually downloading the files. However, even with all the files available locally, replay can take a very long time.
