Specifically, I'm interested in knowing what the state of inflation votes were at an arbitrary historic timestamp. I can run the ledger backward to figure balances, but I'm not sure how I'd figure out previous votes if someone changes their vote.
In order to get the historical inflation votes data you need to run a Horizon node with
CATCHUP_COMPLETE parameter set to
Once a fully synced instance is ready, you will be able to query
history_effects table which contains all ledger effects. Inflation destination is set by
SET_OPTIONS operation that yields
SIGNER_UPDATED effect despite the fact that official documentation mentions
Account Home Domain Updated effect. Here is a bug description.
Therefore in your case an SQL statement that retrieves all
inflation_dest changes will look like this:
SELECT * FROM history_effects WHERE type=12 AND details->>'inflation_dest' IS NOT NULL ORDER BY history_operation_id, order
Add a filter by date and aggregate results to get
inflation_dest for all accounts at a given point in time.