1

Disk usage is growing steadily and we cannot find the solution. There are two questions from about 2-3 years ago:

Stop Horizon/Stellar node growing disk usage

Pruning and keeping disk usage low

But in neither of them is the solution clear.

Our core setting:

CATCHUP_RECENT=100
AUTOMATIC_MAINTENANCE_PERIOD=60
CATCHUP_COMPLETE=false

Postgresql settings:

# AUTOVACUUM PARAMETERS
#------------------------------------------------------------------------------

autovacuum = on                 # Enable autovacuum subprocess?  'on'
                                        # requires track_counts to also be on.
#log_autovacuum_min_duration = -1       # -1 disables, 0 logs all actions and
                                        # their durations, > 0 logs only
                                        # actions running at least this number
                                        # of milliseconds.
autovacuum_max_workers = 2              # max number of autovacuum subprocesses
                                        # (change requires restart)
autovacuum_naptime = 1          # time between autovacuum runs
autovacuum_vacuum_threshold = 50        # min number of row updates before
                                        # vacuum
#autovacuum_analyze_threshold = 50      # min number of row updates before
                                        # analyze
autovacuum_vacuum_scale_factor = 0.01   # fraction of table size before vacuum
#autovacuum_analyze_scale_factor = 0.01 # fraction of table size before analyze
#autovacuum_freeze_max_age = 200000000  # maximum XID age before forced vacuum
                                        # (change requires restart)
#autovacuum_multixact_freeze_max_age = 400000000        # maximum multixact age
                                        # before forced vacuum
                                        # (change requires restart)
autovacuum_vacuum_cost_delay = 20ms     # default vacuum cost delay for
                                        # autovacuum, in milliseconds;
                                        # -1 means use vacuum_cost_delay
autovacuum_vacuum_cost_limit = 1000     # default vacuum cost limit for
                                        # autovacuum, -1 means use
                                        # vacuum_cost_limit

Ledger history count in horizon.env is set to and not growing (actual count stabilized and when checked is ~18500):

HISTORY_RETENTION_COUNT="17280"

Autovacuum workers seem to be working, but the disk usage is still growing by several GB daily.

1 Answer 1

0

What is AUTOMATIC_MAINTENANCE_COUNT set to in core?

Can you check on the sizes of individual tables? Like if you connect to postgres using psql, what do you see for select count(*) from txhistory or select count(*) from publishqueue?

When you say "disk usage" is increasing, are you sure it is postgres disk usage and not (say) archive disk usage or bucket disk usage? Which directories are growing and what are their current sizes?

1
  • Thank you for trying to help! We have reinstalled again stellar about 5 days ago, as the disk was full. Interestingly, it seems there is no disk usage growth this time, which is puzzling. AUTOMATIC_MAINTENANCE_COUNT is set to 60 (min) (previously experimented also with 10 and 1 min) Two psql queries do not return any value. We keep monitoring and I will update if the database starts growing again. Thanks!
    – hotbit
    Aug 13, 2021 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.