Add ESACCI biomass benchmark (2015-2024) and observation-error test - #4064
Merged
Conversation
Adds the ESA CCI Biomass v7.0 product as a second, more recent (2015-2024) aboveground-biomass benchmark with a per-pixel uncertainty layer. Includes a converter to ILAMB-compatible netCDF (regridded to the XuSaatchi grid, dry biomass converted to carbon with the IPCC 0.47 fraction), a scoring and observation-error module that adds observation error to the ensemble spread, a notebook, tests, and figures. The ensemble is equally overconfident against XuSaatchi, ESACCI 2020, and ESACCI 2024, and the overconfidence persists after accounting for observation error.
mdietze
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the ESACCI Biomass v7.0 product as a second, more recent (2015-2024) aboveground-biomass benchmark with a per-pixel uncertainty layer. Includes a converter to ILAMB-compatible netCDF (regridded to the XuSaatchi grid, dry biomass converted to carbon with the IPCC 0.47 fraction), a scoring and observation-error module that adds observation error to the ensemble spread, tests, and figures. The ensemble is equally overconfident against XuSaatchi, ESACCI 2020, and ESACCI 2024, and the overconfidence persists after accounting for observation error.
Description
Extends the biomass benchmarking in
inst/ilamb/to use ESA CCI Biomass v7.0 as a second, independent above-ground biomass benchmark alongside the existing XuSaatchi product, and adds an observation-error test to the ensemble calibration assessment.Contents:
convert_esacci_to_ilamb.pyconverts the ESACCI aggregated GeoTIFF to an ILAMB-compatible netCDF on the XuSaatchi grid. It extracts the requested years, regrids 0.25 to 0.5 degree, and applies the IPCC woody carbon fraction (0.47) to convert dry biomass to carbon density, since XuSaatchi and the model output are in carbon. It writes abiomassvariable and abiomass_sdper-pixel uncertainty variable. The geospatial imports are lazy so the constants can be imported without the geospatial stack installed.score_esacci.pysamples the downscaled ensemble at the assimilation sites and computes the calibration diagnostics against ESACCI and XuSaatchi, then adds observation error to the ensemble spread and re-tests. Itscoverage_with_obs_errorandspread_skill_with_obs_errorfunctions are general and apply to any benchmark that carries an uncertainty estimate.esacci_figures.pyproduces three figures: the ESACCI-versus-XuSaatchi comparison, the calibration against all three benchmarks, and the observation-error result.test_esacci.pycovers the year-to-band mapping, the carbon conversion, and the observation-error diagnostics, using only NumPy and the calibration module.README_esacci.mddocuments the data source (CEDA), the units conversion, and how to run everything.Results at the assimilation sites: the ensemble is equally overconfident against XuSaatchi (~2010), ESACCI 2020, and ESACCI 2024 (spread-skill ratio ~0.07 in all three), so the finding is not specific to an older benchmark. Accounting for observation error does not rescue the calibration: even when the observation error is estimated generously from the difference between the XuSaatchi and ESACCI products, coverage rises only from ~0.02 to ~0.10 and the ratio barely moves, both far below their calibrated targets.
This builds on the ensemble calibration diagnostics already in
inst/ilamb/and importsensemble_calibration.pyfrom that addition, so the per-region and per-benchmark numbers are computed consistently with the domain-wide ones.Motivation and Context
The existing above-ground biomass benchmark (XuSaatchi) is frozen near 2010 and carries no per-pixel uncertainty, while the reanalysis runs through 2024. ESACCI provides annual maps spanning the reanalysis period with an uncertainty layer, which makes it possible to (1) check whether the ensemble overconfidence is specific to the older benchmark by scoring against contemporary 2020 and 2024 observations, and (2) propagate observation error into the calibration to test whether the overconfidence survives when the observation is not treated as perfect. Both checks are addressed here.
Review Time Estimate
Types of changes
Checklist: