Skip to content

Commit a08bffc

Browse files
committed
Document config fields and benchmark customization (review feedback for #4038)
Add inline documentation to pecan_ilamb.cfg explaining every confrontation field (variable, weight at both levels, mass_weighting, skip flags, units), and a Customizing the benchmark section to the README covering how to update or add models, add benchmark datasets, and adjust scoring weights. Addresses review feedback on the PR.
1 parent 3a6f978 commit a08bffc

2 files changed

Lines changed: 94 additions & 0 deletions

File tree

modules/benchmark/inst/ilamb/README.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -254,3 +254,45 @@ figure.
254254
- TRENDY: Global Carbon Budget 2024, public download index.
255255
- Benchmarks: Xu & Saatchi 2021 (biomass), GIMMS LAI4g (leaf area index),
256256
HWSD2 (soil carbon).
257+
258+
## Customizing the benchmark
259+
260+
The benchmark is designed to stay usable as models and datasets evolve. Three
261+
common changes: updating or adding models, adding benchmark datasets, and
262+
adjusting the scoring weights. The per-field meaning of every configuration
263+
option is documented inline in `pecan_ilamb.cfg`.
264+
265+
### Updating or adding a model
266+
267+
Each model is just a directory under the model root (for example
268+
`ilamb_models_2012_2014/`), containing `cVeg.nc`, `cSoil.nc`, and `lai.nc` on
269+
the analysis grid. ILAMB scores every model directory it finds there, so:
270+
271+
- To add a model, place its directory (with the three variable files, regridded
272+
to the analysis grid) under the model root and re-run `ilamb-run`.
273+
- To update a model to a newer version (for example a new CMIP6 or TRENDY
274+
release), regenerate its files with the relevant build script and overwrite
275+
that model's directory. The build scripts take the source and output paths as
276+
arguments, so pointing them at a new data release does not require code edits.
277+
278+
The ensemble means (CMIP6, TRENDY, PEcAn) are rebuilt from the individual
279+
members by `build_window_ensembles.py`, so re-run it after changing the set of
280+
member models so the means reflect the new set.
281+
282+
### Adding a benchmark dataset
283+
284+
Benchmarks are defined in `pecan_ilamb.cfg`. To add one for an existing
285+
variable, copy the dataset block under the relevant `[h2: ...]` variable, rename
286+
it, and point its `source` at the new benchmark netCDF (placed under
287+
`ILAMB_ROOT/DATA/...`). When more than one benchmark is listed under a variable,
288+
their `weight` values set how they combine. To score a new variable entirely,
289+
add a new `[h2: ...]` block and make sure the model files contain that variable.
290+
291+
### Adjusting weights
292+
293+
Scoring weights live in `pecan_ilamb.cfg` at two levels: the `weight` on each
294+
`[h2: ...]` variable sets how much that variable counts toward the overall
295+
score relative to the other variables, and the `weight` on each dataset block
296+
sets how much that dataset counts within its variable. Both are relative
297+
weights, not percentages. See the inline comments in the config for the full
298+
per-field reference.

modules/benchmark/inst/ilamb/pecan_ilamb.cfg

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,58 @@
88
# - LAI: GIMMS_LAI4g (1982-2020). Direct lai comparison.
99
# - Soil carbon: HWSD2 (time-invariant). PEcAn TotSoilCarb is 0-200cm vs
1010
# benchmark shallower; PEcAn reads higher. Flagged for review.
11+
#
12+
# ---------------------------------------------------------------------------
13+
# How this file is structured (ILAMB confrontation config)
14+
# ---------------------------------------------------------------------------
15+
# [h1: ...] A top-level section header that groups variables on the
16+
# scorecard. Purely organizational; "bgcolor" sets its row color.
17+
#
18+
# [h2: ...] A variable to be scored (one carbon-cycle quantity). Its fields:
19+
# variable The ILAMB/CMOR variable name ILAMB looks for in the
20+
# benchmark file (e.g. "lai", "cSoil"). For "biomass" the
21+
# model files store "cVeg", so alternate_vars maps it.
22+
# alternate_vars Other variable name(s) to accept from the model files if
23+
# "variable" is not found (here: model cVeg -> biomass).
24+
# weight How much this variable counts toward the overall score,
25+
# RELATIVE to the other [h2] variables. These are relative
26+
# weights, not percentages: Biomass 5, LAI 3, Soil 5 means
27+
# biomass and soil each count 5/13 and LAI 3/13 of the total.
28+
# Raise a variable's weight to make it matter more overall.
29+
# mass_weighting If True, spatial errors are weighted by the magnitude of
30+
# the field, so high-carbon regions count more than near-zero
31+
# ones. Standard for carbon pools.
32+
# cmap Colormap for this variable's plots (cosmetic only).
33+
#
34+
# [BenchmarkName] An observational dataset to compare against, listed
35+
# directly under its [h2] variable. Its fields:
36+
# source Path to the benchmark netCDF, relative to ILAMB_ROOT
37+
# (set via the ILAMB_ROOT environment variable). This is the
38+
# observation the models are scored against.
39+
# weight How much THIS dataset counts within its variable, relative
40+
# to any other benchmark datasets listed under the same [h2].
41+
# With a single dataset per variable (as here) its value only
42+
# matters if you add a second benchmark for that variable.
43+
# table_unit Unit shown in the summary score tables (e.g. "Pg").
44+
# plot_unit Unit used in the map plots (e.g. "kg m-2").
45+
# space_mean If True, report a spatial mean; if False, report a spatial
46+
# integral/sum. False suits total-pool comparisons.
47+
# skip_rmse Skip the RMSE component of the score. Used where an
48+
# absolute-value RMSE is not meaningful for this comparison
49+
# (e.g. pool-definition mismatches noted above).
50+
# skip_cycle Skip the seasonal-cycle score component. Used for LAI here
51+
# because the benchmark and model sampling differ seasonally.
52+
#
53+
# ---------------------------------------------------------------------------
54+
# To ADD A NEW BENCHMARK for an existing variable: copy a [BenchmarkName]
55+
# block under the relevant [h2], rename it, and point "source" at the new
56+
# netCDF (placed under ILAMB_ROOT/DATA/...). Multiple benchmarks under one
57+
# [h2] are combined using their "weight" values.
58+
#
59+
# To ADD A NEW VARIABLE: add a new [h2: ...] block with its "variable" name
60+
# and at least one [BenchmarkName] dataset beneath it, then make sure the
61+
# model files contain that variable (see build scripts and README).
62+
# ---------------------------------------------------------------------------
1163

1264
[h1: Ecosystem and Carbon Cycle]
1365
bgcolor = "#ECFFE6"

0 commit comments

Comments
 (0)