horizon.stellar.org | 504: Gateway time-out errors are FREQUENT!

Question

I’ve been using the py-stellar-base library to access the stellar public horizon server (version from late last year) and up till recently it’s been fine.

Now I get timeouts (no response from horizon) all the time…. but I’m running the same code as always. So if I run 10 requests, there is a pretty good chance that I'll timeout on one of them. In short, the reliability is complete unacceptable.

Is there something wrong at my end or is the public horizon server unreliable? I looked at the https://horizon.stellar.org/metrics, but I can’t figure out how to interpret the fields…. Are the failures because of user made a mistake or because Horizon failed to respond? What’s the right metric to look at?

Here's what the python output stream looks like when I do a simple add a trustline request. It seems like stellar is using Cloudflare and it is getting a gateway timeout error, so it looks like a Cloudflare->Stellar issue.

    >>> app.add_trustline(JOD)
====Start: add trustline
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
WARNING:root:Submit retry attempt 2
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
WARNING:root:Submit retry attempt 1
WARNING:root:Horizon submit exception: Expecting value: line 1 column 1 (char 0), reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>




</head>
<body>
<div id="cf-wrapper">



    <div id="cf-error-details" class="cf-error-details-wrapper">
        <div class="cf-wrapper cf-error-overview">
            <h1>

              <span class="cf-error-type">Error</span>
              <span class="cf-error-code">504</span>
              <small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 &bull; 2019-02-17 05:17:01 UTC</small>
            </h1>
            <h2 class="cf-subheadline">Gateway time-out</h2>
        </div><!-- /.error-overview -->

        <div class="cf-section cf-highlight cf-status-display">
            <div class="cf-wrapper">
                <div class="cf-columns cols-3">

<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">San Jose</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">horizon.stellar.org</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

                </div>

            </div>
        </div><!-- /.status-display -->

        <div class="cf-section cf-wrapper">
            <div class="cf-columns two">
                <div class="cf-column">
                    <h2>What happened?</h2>
                    <p>The web server reported a gateway time-out error.</p>
                </div>

                <div class="cf-column">
                    <h2>What can I do?</h2>
                    <p>Please try again in a few minutes.</p>
                </div>
            </div>

        </div><!-- /.section -->

        <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>4aa5c8d56ff16be0</strong></span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Your IP</span>: 99.104.205.249</span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

WARNING:root:Submit retry attempt 0
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11), reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>




</head>
<body>
<div id="cf-wrapper">



    <div id="cf-error-details" class="cf-error-details-wrapper">
        <div class="cf-wrapper cf-error-overview">
            <h1>

              <span class="cf-error-type">Error</span>
              <span class="cf-error-code">504</span>
              <small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 &bull; 2019-02-17 05:17:01 UTC</small>
            </h1>
            <h2 class="cf-subheadline">Gateway time-out</h2>
        </div><!-- /.error-overview -->

        <div class="cf-section cf-highlight cf-status-display">
            <div class="cf-wrapper">
                <div class="cf-columns cols-3">

<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">San Jose</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">horizon.stellar.org</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

                </div>

            </div>
        </div><!-- /.status-display -->

        <div class="cf-section cf-wrapper">
            <div class="cf-columns two">
                <div class="cf-column">
                    <h2>What happened?</h2>
                    <p>The web server reported a gateway time-out error.</p>
                </div>

                <div class="cf-column">
                    <h2>What can I do?</h2>
                    <p>Please try again in a few minutes.</p>
                </div>
            </div>

        </div><!-- /.section -->

        <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>4aa5c8d56ff16be0</strong></span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Your IP</span>: 99.104.205.249</span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->


    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

Traceback (most recent call last):
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse()
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1321, in getresponse
    response.begin()
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1049, in recv_into
    return self.read(nbytes, buffer)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 908, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 367, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
    raise value
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 306, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\horizon.py", line 141, in submit
    url, data=params, timeout=self.request_timeout)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 572, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 524, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 637, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#0>", line 1, in <module>
    app.add_trustline(JOD)
  File "C:\Users\stk\Documents\GitHub\tokenx-stellar-python\account.py", line 178, in add_trustline
    t.submit()
  File "C:\Users\stk\Documents\GitHub\tokenx-stellar-python\account.py", line 316, in submit
    response=self.b.submit()
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\builder.py", line 779, in submit
    return self.horizon.submit(self.gen_xdr())
  File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\horizon.py", line 155, in submit
    reply.status_code, reply.text), reply.status_code)
stellar_base.exceptions.HorizonError: Invalid horizon reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>


<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>




</head>
<body>

<div id="cf-error-details" class="cf-error-details-wrapper">
    <div class="cf-wrapper cf-error-overview">
        <h1>

          <span class="cf-error-type">Error</span>
          <span class="cf-error-code">504</span>
          <small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 &bull; 2019-02-17 05:17:01 UTC</small>
        </h1>
        <h2 class="cf-subheadline">Gateway time-out</h2>
    </div><!-- /.error-overview -->

    <div class="cf-section cf-highlight cf-status-display">
        <div class="cf-wrapper">
            <div class="cf-columns cols-3">

You Browser Working San Jose Cloudflare Working horizon.stellar.org Host Error

            </div>

        </div>
    </div><!-- /.status-display -->

    <div class="cf-section cf-wrapper">
        <div class="cf-columns two">
            <div class="cf-column">
                <h2>What happened?</h2>
                <p>The web server reported a gateway time-out error.</p>
            </div>

            <div class="cf-column">
                <h2>What can I do?</h2>
                <p>Please try again in a few minutes.</p>
            </div>
        </div>

    </div><!-- /.section -->

    <div class="cf-error-footer cf-wrapper">

Cloudflare Ray ID: 4aa5c8d56ff16be0 • Your IP: 99.104.205.249 • Performance & security by Cloudflare

</div><!-- /#cf-error-details -->

sui · Accepted Answer · 2019-02-17 12:04:32Z

504 means that a proxy server between you and the horizon server could connect to horizon but did not get a response in time.

The /metrics shows completely different values on every reload. So actually they run a bunch of horizon servers behind a load balancer, you are connected with a different server each time, the one that causes 504 is supposedly under heavy load and can't respond to all requests in a timeframe that the load balancer expects it to.

I'd recommend to run your own nodes.

Stack Exchange Network

current community

your communities

more stack exchange communities

horizon.stellar.org | 504: Gateway time-out errors are FREQUENT!

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged horizon python-sdk or ask your own question.

Hot Network Questions

horizon.stellar.org | 504: Gateway time-out errors are FREQUENT!

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged horizon python-sdk or ask your own question.

Hot Network Questions