I’ve been using the py-stellar-base library to access the stellar public horizon server (version from late last year) and up till recently it’s been fine.
Now I get timeouts (no response from horizon) all the time…. but I’m running the same code as always. So if I run 10 requests, there is a pretty good chance that I'll timeout on one of them. In short, the reliability is complete unacceptable.
Is there something wrong at my end or is the public horizon server unreliable? I looked at the https://horizon.stellar.org/metrics, but I can’t figure out how to interpret the fields…. Are the failures because of user made a mistake or because Horizon failed to respond? What’s the right metric to look at?
Here's what the python output stream looks like when I do a simple add a trustline request. It seems like stellar is using Cloudflare and it is getting a gateway timeout error, so it looks like a Cloudflare->Stellar issue.
>>> app.add_trustline(JOD)
====Start: add trustline
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
WARNING:root:Submit retry attempt 2
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
WARNING:root:Submit retry attempt 1
WARNING:root:Horizon submit exception: Expecting value: line 1 column 1 (char 0), reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
</head>
<body>
<div id="cf-wrapper">
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-error-overview">
<h1>
<span class="cf-error-type">Error</span>
<span class="cf-error-code">504</span>
<small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 • 2019-02-17 05:17:01 UTC</small>
</h1>
<h2 class="cf-subheadline">Gateway time-out</h2>
</div><!-- /.error-overview -->
<div class="cf-section cf-highlight cf-status-display">
<div class="cf-wrapper">
<div class="cf-columns cols-3">
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-browser"></i>
<i class="cf-icon-status cf-icon-ok"></i>
</div>
<span class="cf-status-desc">You</span>
<h3 class="cf-status-name">Browser</h3>
<span class="cf-status-label">Working</span>
</div>
<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-cloud"></i>
<i class="cf-icon-status cf-icon-ok"></i>
</div>
<span class="cf-status-desc">San Jose</span>
<h3 class="cf-status-name">Cloudflare</h3>
<span class="cf-status-label">Working</span>
</div>
<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-server"></i>
<i class="cf-icon-status cf-icon-error"></i>
</div>
<span class="cf-status-desc">horizon.stellar.org</span>
<h3 class="cf-status-name">Host</h3>
<span class="cf-status-label">Error</span>
</div>
</div>
</div>
</div><!-- /.status-display -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2>What happened?</h2>
<p>The web server reported a gateway time-out error.</p>
</div>
<div class="cf-column">
<h2>What can I do?</h2>
<p>Please try again in a few minutes.</p>
</div>
</div>
</div><!-- /.section -->
<div class="cf-error-footer cf-wrapper">
<p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>4aa5c8d56ff16be0</strong></span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Your IP</span>: 99.104.205.249</span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Performance & security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
</div><!-- /.error-footer -->
</div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>
WARNING:root:Submit retry attempt 0
WARNING:root:Horizon submit exception: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11), reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
</head>
<body>
<div id="cf-wrapper">
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-error-overview">
<h1>
<span class="cf-error-type">Error</span>
<span class="cf-error-code">504</span>
<small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 • 2019-02-17 05:17:01 UTC</small>
</h1>
<h2 class="cf-subheadline">Gateway time-out</h2>
</div><!-- /.error-overview -->
<div class="cf-section cf-highlight cf-status-display">
<div class="cf-wrapper">
<div class="cf-columns cols-3">
<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-browser"></i>
<i class="cf-icon-status cf-icon-ok"></i>
</div>
<span class="cf-status-desc">You</span>
<h3 class="cf-status-name">Browser</h3>
<span class="cf-status-label">Working</span>
</div>
<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-cloud"></i>
<i class="cf-icon-status cf-icon-ok"></i>
</div>
<span class="cf-status-desc">San Jose</span>
<h3 class="cf-status-name">Cloudflare</h3>
<span class="cf-status-label">Working</span>
</div>
<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
<div class="cf-icon-error-container">
<i class="cf-icon cf-icon-server"></i>
<i class="cf-icon-status cf-icon-error"></i>
</div>
<span class="cf-status-desc">horizon.stellar.org</span>
<h3 class="cf-status-name">Host</h3>
<span class="cf-status-label">Error</span>
</div>
</div>
</div>
</div><!-- /.status-display -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2>What happened?</h2>
<p>The web server reported a gateway time-out error.</p>
</div>
<div class="cf-column">
<h2>What can I do?</h2>
<p>Please try again in a few minutes.</p>
</div>
</div>
</div><!-- /.section -->
<div class="cf-error-footer cf-wrapper">
<p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>4aa5c8d56ff16be0</strong></span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Your IP</span>: 99.104.205.249</span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Performance & security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
</div><!-- /.error-footer -->
</div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>
Traceback (most recent call last):
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "<string>", line 2, in raise_from
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 1321, in getresponse
response.begin()
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 296, in begin
version, status, reason = self._read_status()
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\http\client.py", line 257, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 1049, in recv_into
return self.read(nbytes, buffer)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\ssl.py", line 908, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\util\retry.py", line 367, in increment
raise six.reraise(type(error), error, _stacktrace)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\packages\six.py", line 686, in reraise
raise value
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
chunked=chunked)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\urllib3\connectionpool.py", line 306, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\horizon.py", line 141, in submit
url, data=params, timeout=self.request_timeout)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 572, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 524, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\sessions.py", line 637, in send
r = adapter.send(request, **kwargs)
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\requests\adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='horizon.stellar.org', port=443): Read timed out. (read timeout=11)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
app.add_trustline(JOD)
File "C:\Users\stk\Documents\GitHub\tokenx-stellar-python\account.py", line 178, in add_trustline
t.submit()
File "C:\Users\stk\Documents\GitHub\tokenx-stellar-python\account.py", line 316, in submit
response=self.b.submit()
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\builder.py", line 779, in submit
return self.horizon.submit(self.gen_xdr())
File "C:\Users\stk\AppData\Local\Programs\Python\Python37-32\lib\site-packages\stellar_base\horizon.py", line 155, in submit
reply.status_code, reply.text), reply.status_code)
stellar_base.exceptions.HorizonError: Invalid horizon reply: [504] <!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>horizon.stellar.org | 504: Gateway time-out</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>
</head>
<body>
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-error-overview">
<h1>
<span class="cf-error-type">Error</span>
<span class="cf-error-code">504</span>
<small class="heading-ray-id">Ray ID: 4aa5c8d56ff16be0 • 2019-02-17 05:17:01 UTC</small>
</h1>
<h2 class="cf-subheadline">Gateway time-out</h2>
</div><!-- /.error-overview -->
<div class="cf-section cf-highlight cf-status-display">
<div class="cf-wrapper">
<div class="cf-columns cols-3">
</div>
</div>
</div><!-- /.status-display -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2>What happened?</h2>
<p>The web server reported a gateway time-out error.</p>
</div>
<div class="cf-column">
<h2>What can I do?</h2>
<p>Please try again in a few minutes.</p>
</div>
</div>
</div><!-- /.section -->
<div class="cf-error-footer cf-wrapper">
Cloudflare Ray ID: 4aa5c8d56ff16be0 • Your IP: 99.104.205.249 • Performance & security by Cloudflare
</div><!-- /#cf-error-details -->
