1

We faced the problem when horizonclient returned errors while sending an envelope but transaction has been created. Please bash my thoughts and tell me where I'm wrong and how you do that.

The process we implemented when creating transactions looks as follows (some details omitted keeping the logic):

func createTransaction(){
  env := loadEnvelopeFromStorage()
  txHash, err := sendEnvelope(env) // 1
  if err == nil {
    storeTxHash(txHash)
    deleteEnvelope()
    return nil
  }

  if err == ErrTooLate || err == ErrBadSeq { // 2
    env := rebuildEnvelope()
    storeEnvelopeInStorage(env)
    createTransaction()
    return nil
  } else {
    return error
  }
}

I intentionally don't write any synchronisation code and we can assume that this code works as in a single-thread application on one machine only.

Our assumptions about sending envelopes are:

  1. If an envelope has already been sent and accepted then a transaction hash will be returned with no error along even if the sequence number of an envelope is less then the account sequence number.
  2. If an envelope has a sequence number less or equal then account sequence number and the envelope hasn't been successfully sent, then bad_seq error is returned.
  3. If an envelope has a time bound with the higher value less then timestamp of the moment of sending the envelope, then too_late error is returned.
  4. Any other error can be returned.

In case of 2) and 3) we should forget about the envelope, build another one and try to send it. Out experiments and tests assured that in case of 2) and 3) a new transaction is not created and no duplication happens. We tested that when parallel sending a certain envelope happens then the transaction creates only ones. Moreover, we tested that when we send a envelope that had already been sent before some more transaction were send then the transaction hash of the first transaction generated by this envelope is returned.

But what we've observed recently was the following: We received a deposit from a customer and had to transfer funds from our stellar address to another. Here is the log with timestamps:

01 08:04:48 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] could not send envelope: result_codes not populated: horizon error: "Timeout" - check horizon.Error.Problem for more information
02 08:04:48 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] not rebulding envelope
03 08:05:48 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] could not send envelope: EOF
04 08:05:48 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] not rebulding envelope
05 08:06:56 Got payment 7a51....f703 on stellar address GAM2...YKG6
06 08:09:07 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] could not send envelope: stellar: need to rebuild tx: bad sequence
07 08:09:07 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] rebuilding/resending envelope
08 08:09:13 [tx.id=cbc4d9b4-6cd0-4180-b85c-58e3acd6b933] is successfully sent to sent to stellar: hash=dc95....02ae
09 08:09:13 Got payment dc95....02ae on stellar address GAM2...YKG6
  • Lines 01-02: We tried to send, failed because of timeout, log that we don't rebuild envelope because of this error
  • Lines 03-04: In a minute we tried again, failed because of EOF, don't rebuild envelope
  • Line 05: We got a payment from the stream of the destination account
  • Line 06-08: We came back to the envelope and tried to send it again. Error bad_seq received what indicates that some payment has been made before the envelope and the moment of line 06. Bebuild, resend, success?!.
  • Line 09: We got a payment from the stream of the destination account. Duplicate.

So my question is, why we got bad_seq error despite the fact that transaction has been applied and the destination account has been notified about that?

I base my logic and code on what documentation says: If the transaction has already been successfully applied to the ledger, horizon will simply return the saved result and not attempt to submit the transaction again. Is there a bug in my logic or in horizon?

6
  • Did you use horizon.stellar.org or your own Horizon? If it was your own, what was the Horizon version? Can you share tx hash? – Bartek Nowotarski Feb 1 at 14:10
  • This is horizon.stellar.org. tx hash is: 1) 348ee728fabdc8b154713ba95d85dbc618f4f5f269f738d6d7c99835d3190b75 - EOF error 2) e71cd4151533b3448e90ea13a37a501335b73db1b652f5074d77bc921d8f55a4 - duplicate Go log: EOF error decoding horizon.Problem github.com/stellar/go/support/errors.Wrap /go/src/app/vendor/github.com/stellar/go/support/errors/main.go:43 github.com/stellar/go/clients/horizonclient.decodeResponse /go/src/app/vendor/github.com/stellar/go/clients/horizonclient/internal.go:30 github.com/stellar/go/clients/horizonclient.(*Client).sendRequestURL – gekpp Feb 1 at 14:43
  • Thanks! For 348ee I can see timeout error and e71cd4 there was 200 OK response with transaction details. But according to the logic you explained you should not increment seqnum in this case (you got 2x EOF). Also, the hashed in the log in your original message are different (7a51 and dc95). Are you sure this is the same issue? – Bartek Nowotarski Feb 1 at 15:18
  • Sorry, the hashes are different because in the comment i provided hashes from the issue I faced today, not on the day I posted the initial message. The initial hashes were 7a517b13153a36444f0b24029b1a2b6c142e89b36906eecc3f830553c242f703 and dc95a7d9af9f3fa7b8e4155b33d1231d1500fd7d5edefa320920c70b27e102ae . The question is whether my logic correct or not? If i get bad sequence error then I rebuild an envelope and send a new one, because otherwise I would get a transaction hash and proceed without error. What I get is: for the same envelope 1) EOF, 2) bad_seq. That's why i rebuild. – gekpp Feb 1 at 16:56
  • as I mentioned above I don't see bad sequence error for 348ee or e71cd4 in our Horizon logs. That's why I asked if this was the same issue? Regarding 7a517b and dc95a7d I don't have logs for it in the short-term storage, will check with ops team. – Bartek Nowotarski Feb 2 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.