Add met2CF.ERA5.reanalysis: Convert ERA5 Reanalysis Data to CF-Compliant NetCDF

[divine7022]

Description

This PR introduces a new function, met2CF.ERA5.reanalysis, to the PEcAn data atmosphere module for processing ERA5 reanalysis NetCDF files into CF-compliant format, and is designed specifically for non-ensemble ("reanalysis") data products.

Motivation and Context

ERA5 reanalysis data serves as a fallback when ensembling, for example, in cases where AmeriFlux meteorological data is unavailable or incomplete.

Review Time Estimate

• Immediately
• Within one week
• When possible

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My change requires a change to the documentation.
• My name is in the list of CITATION.cff
• I agree that PEcAn Project may distribute my contribution under any or all of
  • the same license as the existing code,
  • and/or the BSD 3-clause license.
• I have updated the CHANGELOG.md.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.
• All new and existing tests passed.

[mdietze]

Conceptually, how is this different from the existing met2CF.ERA5 and why are we adding a second met2CF function for the same product? If there was something wrong with the existing function, wouldn't it be better to fix it than create a duplicate? How would users know which function to use?

[divine7022]

@mdietze that’s a great point!
The current met2CF.ERA5 function is specifically designed to process the ERA5 native 10 member ensemble product. It is tightly coupled to the structure and logic (multi-member handling, naming, looping, and metadata) of the ensemble dataset.

However, ERA5 Reanalysis is a single-member dataset rather than an ensemble.
To support both types, the download.ERA5_cds function has now been made more flexible, allowing it to handle both ensemble and reanalysis downloads (#3547)

Motivation and Context

ERA5 reanalysis data serves as a fallback when ensembling, for example, in cases where AmeriFlux meteorological data is unavailable or incomplete.

[mdietze]

Ok, but the existing ERA5 function IS processing the reanalysis. The reanalysis is fundamentally a 10-member ensemble. Yes, one can also download and process the mean of those 10 ensemble members, but why does that require an entirely separate function? Why wouldn't the existing function work for the special case of an ensemble with n=1 members (i.e. the mean file)?

[divine7022]

Thank you for this important question, @mdietze. And I got caught up with some work and wasn’t able to get to it sooner.

after again re viewing the code and considering the implications, I believe the separate function approach is justified for several reasons:

there are several fundamental structural and technical differences between ERA5 HRES (High Resolution) reanalysis and EDA (Ensemble Data Assimilation) that necessitate separate processing functions

https://nordata.physics.utoronto.ca/en/annexes-datasets-descriptions/71-temperature/ecmwf-reanalysis-5th-generation-era5-atmospheric-reanalysis/

can also download and process the mean of those 10 ensemble members, but why does that require an entirely separate function?

since ecwmf now directly provides reanalysis data (HRES), so I think it's not ideal to compute the mean from the 10 ensemble members. Moreover, the current function does not support calculating the mean across ensemble members or handling the case where n = 1

ERA5 dataset contains one (hourly, 31 km) high resolution realisation (referred to as "reanalysis" or "HRES") and a reduced resolution 10 member ensemble (referred to as "ensemble" or "EDA"). The ERA5 dataset contains one (31 km) high resolution realization (HRES) with hourly temporal resolution, and a reduced resolution (62 km) 10 member ensemble (EDA) with 3-hourly temporal resolution, though it is adjustable now.

the existing extract.nc.ERA5 and met2CF.ERA5 while both are interlinked, process 10 ens members and sends list of xts object to met2CF.ERA5

expects:
level parameter for ensemble member extraction (level = ens)

looping through ensemblesN <- seq(1, 10) # Hardcoded for 10 ensembles
variable extraction with raster::brick(ncfile, varname = vname, level = ens)  # Expects level dimension for ensemble members

• ERA5 HRES reanalysis files don't have a level dimension - they're single-realization files.
• the ensemble function processes 10 separate members and creates 10 separate output folders

results_list <- ensemblesN %>%
  purrr::map(function(i) {
    # Creates separate folder for each ensemble member
    ensemble_folder <- file.path(outfolder, identifier)
    # Processes each member separately
  })

the reanalysis function processes a single dataset and creates one output folder

# single processing path
reanalysis_folder <- file.path(outfolder, identifier)
# no ensemble looping required

• if we considered modifying existing function

conditional logic:
we could add branching logic to the existing function:

if (data_type == "ensemble") {
  # current ensemble processing logic
} else {
  # reanalysis processing logic
}

but problem with this approach:

• the function would become significantly more complex and harder to maintain
• different naming conventions, identifiers, and file structures

Any thoughts or suggestions are appreciated, I’m open to considering them

[mdietze]

I don't think conditional branching is the correct way to handle bringing these two functions together. Just because the current function hard-codes the ensemble size doesn't mean that it has to be hard coded. Similarly, I also don't see why the spatial or temporal resolution would need to be hard coded. I also wasn't saying that we should compute the ensemble mean, but rather that I expect the single layer product to already be the ensemble mean.

[divine7022]

Hey @mdietze, I’ve opened pr #3584 , this update generalizes the functions to handle both ERA5 ensemble and reanalysis formats. previously the code was tailored only for ensembles, but now it flexibly supports both.

If you get a chance, I’d really appreciate it if you could take a look and share any feedback or suggestions.