Fetch obs issues

[james-a-f-ross]

Describe the bug

Some issues have come up when fetching observations, sometimes hangs for 10mins or so then fails and sometimes it works straight away.

could work around this by adding a execution wall time, e.g around 2 mins, then also add retries to the cylc task.

How to reproduce

seems to be random, but by requesting observations

Steps to reproduce the behaviour:

Expected behaviour

should retrieve obs and not hang/fail

[jfrost-mo]

I think this is the same issue as https://github.com/MetOffice/CSET-restricted-files/issues/270.

A retry is sensible but it should probably have a 20 minute retry wait, as Azure MASS can take 15 minutes to stage data from tape to disk. (And rather more currently due to fetch-on-demand issues.)