Add mask fill operator so that bad fill values are set to NaN

[Simon Osborne (mo-sro)]

…to np.nan

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Ensure rose-suite.conf.example has been updated if new diagnostic added.
• Conda lock files have been updated if dependencies have changed.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[github-actions]

Total coverage: 92% (HTML report)

Name                                                              Stmts   Miss Branch BrPart  Cover
---------------------------------------------------------------------------------------------------
src/CSET/__init__.py                                                105      0     14      0   100%
src/CSET/_common.py                                                 156      0     54      0   100%
src/CSET/cset_workflow/app/fetch_fcst/bin/fetch_data.py             117     28     26      0    78%
src/CSET/cset_workflow/app/fetch_nimrod/bin/fetch_nimrod.py          81      8     28     11    83%
src/CSET/cset_workflow/app/finish_website/bin/finish_website.py      79      2      8      2    95%
src/CSET/cset_workflow/app/parbake_recipes/bin/parbake.py            29      0      8      0   100%
src/CSET/cset_workflow/app/send_email/bin/send_email.py              25      0      4      0   100%
src/CSET/cset_workflow/lib/python/jinja_utils.py                     17      0      6      0   100%
src/CSET/extract_workflow.py                                        103      1     26      1    98%
src/CSET/graph.py                                                    44      0     14      0   100%
src/CSET/operators/__init__.py                                       89      0     26      0   100%
src/CSET/operators/_atmospheric_constants.py                          9      0      0      0   100%
src/CSET/operators/_colormaps.py                                    249      3     72      4    98%
src/CSET/operators/_stash_to_lfric.py                                 3      0      0      0   100%
src/CSET/operators/_utils.py                                        200      8     74      6    95%
src/CSET/operators/ageofair.py                                      142      7     64      5    94%
src/CSET/operators/aggregate.py                                      77      1     22      1    98%
src/CSET/operators/aviation.py                                       61      0     18      0   100%
src/CSET/operators/collapse.py                                      155      8     72      3    93%
src/CSET/operators/constraints.py                                   113      7     50      3    93%
src/CSET/operators/convection.py                                     38      4     10      2    88%
src/CSET/operators/ensembles.py                                      27      0     14      0   100%
src/CSET/operators/feature.py                                        44      0     10      0   100%
src/CSET/operators/filters.py                                        67      2     30      0    98%
src/CSET/operators/fluxes.py                                         41      0     10      0   100%
src/CSET/operators/humidity.py                                      135      0     52      0   100%
src/CSET/operators/imageprocessing.py                                57      0     16      0   100%
src/CSET/operators/mesoscale.py                                      18      0      2      0   100%
src/CSET/operators/misc.py                                          210     18     86      3    89%
src/CSET/operators/plot.py                                         1122    158    412     71    82%
src/CSET/operators/power_spectrum.py                                 98      3     30      3    95%
src/CSET/operators/precipitation.py                                 204      2     92      2    99%
src/CSET/operators/pressure.py                                       41      0     12      0   100%
src/CSET/operators/read.py                                          442     22    190     17    94%
src/CSET/operators/regrid.py                                        147      1     70      3    98%
src/CSET/operators/scoreswrappers.py                                217     22     66      5    88%
src/CSET/operators/temperature.py                                   121      0     32      0   100%
src/CSET/operators/transect.py                                       63      0     24      0   100%
src/CSET/operators/wind.py                                           46      3     10      2    91%
src/CSET/operators/write.py                                          15      0      6      0   100%
src/CSET/recipes/__init__.py                                        104      0     28      0   100%
src/CSET/sample_data/__init__.py                                      0      0      0      0   100%
---------------------------------------------------------------------------------------------------
TOTAL                                                              5111    308   1788    144    92%

[ukmo-huw-lewis]

Self-nominated review on this one.

Please document and illustrate what issue this PR is addressing - would typically expect to see linked Issue with evidence of problem, and then evidence of 'problem fixed' in PR.

My default starting point is to propose that we handle fill-value masking within the read operator (e.g. automatic callbacks) rather than require a new operator to be called from recipes (noting maybe only being called for Cardington recipes in first instance?).

Unpacking use-case, with sample data will I hope help us consider best approach to enable the functionality you require.

Can offer more detailed review comment when we have bottomed out these aspects - e.g. how to ensure the hard-coded FillValues do not remove valid data. In general would anticipate FillValue to be identified attribute in file, and then require read operators to handle these appropriately.

Let's play with some real data to unpack use-case.

[Simon Osborne (mo-sro)]

Despite the fact that iris can and does handle masked arrays, e.g. masks that use _FIllValue of 1e10 or 1e11 within the Cardington data netCDFs, these masks somehow get quietly dropped by the time plot.py is reached. No reference to masked arrays appears in plot.py. So when plotting UM vs Cardington data, where the latter contains fill values, these values are being plotted and so the plot axes either auto-correct to absurd limits, or the data doesn't appear if the axes limits are hard-wired to sensible values. Maybe this problem hasn't been encountered before, maybe it's been assumed that masked arrays would be handled across routines without issue. This isn't the case. In addition, putting a function of the kind in this PR in read.py is risky because if certain science calculations are carried out that generate "bad data" from 1e10 or 1e11 data, and masks are not maintained, then this data will never be filtered out before plotting.