Why horizon cursors are so obfuscated?

Question

I wonder if there is any system in values for stellar horizon cursors.
I tried to ask testnet horizon for ledgers, starting from https://horizon-testnet.stellar.org/ledgers/ and then pressing next link. I'm getting cursor values:

42949672960
85899345920
128849018880
171798691840

They seem to be senseless, but if we convert them to binary:

101000000000000000000000000000000000_2, 
1010000000000000000000000000000000000_2, 
1111000000000000000000000000000000000_2, 
10100000000000000000000000000000000000_2,

we can see that most part of them is zero. Definitely, last 32 bits are zero ones. Also. If we convert some number to binary and then append 32 bytes to its tail, we will get a cursor which points to an ledger + 1. Example. Let ledger number will be 100500, then its binary representation is 11000100010010100_2. If we append 32 zero bits to it, and we will get 1100010001001010000000000000000000000000000000000_2, which is 431644213248000_10, then we go to the horizon: https://horizon-testnet.stellar.org/ledgers/?cursor=431644213248000&limit=1&order=asc and see that ledger number is 100501. This phenomenon is not documented anywhere. Could anyone answer why we just can't put there 64-bit long ledger sequence and not confuse everyone who tries to understand what is a horizon cursor? Connected questions, will this method work on other horizon instances? Will it work on other tables?

Answer 1

Cursors are generated using toid package. From package comment:

ID represents the total order of Ledgers, Transactions and
Operations.

Operations within the stellar network have a total order, expressed by three
pieces of information:  the ledger sequence the operation was validated in,
the order which the operation's containing transaction was applied in
that ledger, and the index of the operation within that parent transaction.

We express this order by packing those three pieces of information into a
single signed 64-bit number (we used a signed number for SQL compatibility).

The follow diagram shows this format:

   0                   1                   2                   3
   0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |                    Ledger Sequence Number                     |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
  |     Transaction Application Order     |       Op Index        |
  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

By component:

Ledger Sequence: 32-bits

  A complete ledger sequence number in which the operation was validated.

  Expressed in network byte order.

Transaction Application Order: 20-bits

  The order that the transaction was applied within the ledger it was
  validated.  Accommodates up to 1,048,575 transactions in a single ledger.

  Expressed in network byte order.

Operation Index: 12-bits

  The index of the operation within its parent transaction. Accommodates up
  to 4095 operations per transaction.

  Expressed in network byte order.


Note: API Clients should not be interpreting this value.  We will use it
as an opaque paging token that clients can parrot back to us after having read
it within a resource to page from the represented position in time.

Note: This does not uniquely identify an object.  Given a ledger, it will
share its id with its first transaction and the first operation of that
transaction as well.  Given that this ID is only meant for ordering within a
single type of object, the sharing of ids across object types seems
acceptable.

At the same time it's worth noting that you should not try to decode cursor values as the underlying structure may change at any time without notice.