Do not crash if downloading a tile temporary fails

[Tapter]

When downloading a bunch of tiles to create an mbtiles file, the process is interrupted (program crash) if downloading one tile fails for an unknown reason (probably a temporary server error or something). It would be nice if this error would just be treated transparently by either retrying a few times or ignoring that tile (and reporting that tiles were skipped in the end).

Here's the stacktrace of such an event:

DEBUG:landez.tiles:tile method called with [15, 17144, 11422]
DEBUG:landez.sources:Request WMS tile (15, 17144, 11422)
DEBUG:landez.sources:Download 'https://...'
WARNING:landez.tiles:''
Traceback (most recent call last):
  File "dtk.py", line 30, in <module>
    mb.run()
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 322, in run
    self._gather((z, x, y))
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 374, in _gather
    tilecontent = self.tile((z, x, y))
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/tiles.py", line 172, in tile
    output = self.reader.tile(z, x, y)
  File "/usr/local/lib/python2.7/dist-packages/landez-2.4.0-py2.7.egg/landez/sources.py", line 231, in tile
    f = urllib2.urlopen(request)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
    raise BadStatusLine(line)
httplib.BadStatusLine: ''

[leplatrem]

It is a bug I believe.

We are supposed to retry when the download fails:

landez/landez/sources.py

Lines 172 to 188 in 8df5a9f

	r = DOWNLOAD_RETRIES
	sleeptime = 1
	while r > 0:
	try:
	request = urllib2.Request(url)
	for header, value in self.headers.items():
	request.add_header(header, value)
	stream = urllib2.urlopen(request)
	assert stream.getcode() == 200
	return stream.read()
	except (AssertionError, IOError), e:
	logger.debug(_("Download error, retry (%s left). (%s)") % (r, e))
	r -= 1
	time.sleep(sleeptime)
	# progressivly sleep longer to wait for this tile
	if (sleeptime <= 10) and (r % 2 == 0):
	sleeptime += 1 # increase wait

It looks like this exception is not caught...

[ben-katan]

any progress on this issue?