utils: Add retry logic in shell.download

The `tries` option in wget was not working. The `retry-connrefused`,
'retry-on-host-error' and `retry-on-http-error` options are mandatory
to retry correctly, only with the `tries` option is not enough.

Before:
```
$ wget http://httpbin.org/status/404 --tries=4 --timeout=10.0 --progress=dot:giga
--2021-10-08 11:47:22--  http://httpbin.org/status/404
Resolving httpbin.org (httpbin.org)... 3.209.149.47, 54.156.165.4, 54.159.86.231, ...
Connecting to httpbin.org (httpbin.org)|3.209.149.47|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
2021-10-08 11:47:22 ERROR 404: NOT FOUND.
```

After:
```
$ wget http://httpbin.org/status/404 --tries=4 --retry-connrefused --retry-on-host-error --retry-on-http-error=404,429,503,504 --timeout=10.0 --progress=dot:giga
--2021-10-08 11:48:10--  http://httpbin.org/status/404
Resolving httpbin.org (httpbin.org)... 3.215.162.201, 54.159.86.231, 54.156.165.4, ...
Connecting to httpbin.org (httpbin.org)|3.215.162.201|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:12--  (try: 2)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:14--  (try: 3)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Retrying.

--2021-10-08 11:48:17--  (try: 4)  http://httpbin.org/status/404
Reusing existing connection to httpbin.org:80.
HTTP request sent, awaiting response... 404 NOT FOUND
Giving up.
```

But the `retry-on-http-error` requeres version 1.19.1. This version is
very new and it is not included in all supported OSs.

Final solution was implementing the retry mechanism in cerbero to
support all versions of wget, and also curl and urllib2.

Part-of: <https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1020>

1 files changed, 11 insertions, 7 deletions

diff --git a/cerbero/utils/shell.py b/cerbero/utils/shell.py
index 995ae759..a8fdfd1f 100644
--- a/cerbero/utils/shell.py
+++ b/cerbero/utils/shell.py
@@ -410,13 +410,17 @@ async def download(url, dest, check_cert=True, overwrite=False, logfile=None, mi
     url_fmt = cmd[-1]
     cmd = cmd[:-1]
     for murl in urls:
-        try:
-            return await async_call(cmd + [url_fmt % murl], cpu_bound=False,
-                                    logfile=logfile)
-        except Exception as ex:
-            if os.path.exists(dest):
-                os.remove(dest)
-            errors.append((murl, ex))
+        tries = 2
+        while tries > 0:
+            try:
+                return await async_call(cmd + [url_fmt % murl], cpu_bound=False,
+                                        logfile=logfile)
+            except Exception as ex:
+                if os.path.exists(dest):
+                    os.remove(dest)
+                tries -= 1
+                if tries == 0:
+                    errors.append((murl, ex))
     if len(errors) == 1:
         errors = errors[0]
     raise FatalError('Failed to download {!r}: {!r}'.format(url, errors))