Could max_retries be configurable ?

[thebaptiste]

Is your feature request related to a problem? Please describe.

When an url is down (may be definitively or for a long time), max_retries 500 with retry_after 120s is very long before the request fails...

Describe the solution you'd like

I would like max_retries to be configurable, for example with an optional environment variable such as MULTIURL_MAX_RETRIES

Describe alternatives you've considered

Killing manually the process after 10 ou 20 retries with error messages...

Additional context

I had this problem with climetlab on hurricane dataset :

(...)
Recovering from connection error [HTTPSConnectionPool(host='www.aoml.noaa.gov', port=443): Max retries exceeded with url: /hrd/hurdat/hurdat2.txt (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff8f334aff0>, 'Connection to www.aoml.noaa.gov timed out. (connect timeout=30)'))], attemps 16 of 500
Retrying in 120 seconds
Recovering from connection error [HTTPSConnectionPool(host='www.aoml.noaa.gov', port=443): Max retries exceeded with url: /hrd/hurdat/hurdat2.txt (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff8f334aea0>, 'Connection to www.aoml.noaa.gov timed out. (connect timeout=30)'))], attemps 17 of 500
Retrying in 120 seconds
Recovering from connection error [HTTPSConnectionPool(host='www.aoml.noaa.gov', port=443): Max retries exceeded with url: /hrd/hurdat/hurdat2.txt (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff8f334bdd0>, 'Connection to www.aoml.noaa.gov timed out. (connect timeout=30)'))], attemps 18 of 500
(...)

Organisation

Meteo-France

[swdmike]

Agreed, 500 retries is excessive.