Phase 4: Biogeochemistry MVP Integration and Initial Benchmarks

[divine7022]

Goal

Run the toolkit end to end on real benchmark data, refine the data intake on the friction that real datasets expose, and add integration tests. Phases 1 to 3 built the intake, alignment, metric registry, and Quarto reporting layer (#4017, #4032, #4040, #4041). Phase 4 proves that pipeline on real model output against real observations.

Current status

• The pipeline runs end to end on the Salinas SOC ensemble: ingest ensemble_output.csv(agitated , align monthly model output to the annual observations with align_data(), compute ensemble mean metrics (RMSE, bias, R2), and render the ribbon with observation error bars. Remaining there: generate the Quarto HTML scorecard and add the ensemble spread metrics.
• The pipeline also runs on a real AmeriFlux observation CSV, but paired with a synthetic model series (a smoke test), so its scorecard numbers are not model skill yet.
• The seam with the MAGiC/downscaling side is one EFI-long ensemble_output.csv (datetime, site_id, parameter, variable, prediction, monthly). The toolkit consumes that CSV; the model side is produced upstream, as it was done for Salinas in this pass.

Tasks

MVP integration (the core of Phase 4)

• AmeriFlux MVP : pair a real SIPNET ensemble with AmeriFlux observations (e.g. NEE), replacing the synthetic model series, map the columns, align to the model output, and generate the Quarto report, per the workplan. dataset - https://docs.google.com/spreadsheets/d/1DYSbe-YI15idXOlZxxh3IsCeOCFcGaB3/edit?gid=1056740863
• Salinas SOC: from the data aggregated data I passed, finish the end-to-end run through Validation_report.qmd to an HTML scorecard rather than raw PDFs.
• Secondary, once AmeriFlux is proven: a targeted subset of the CH4/N2O meta datasets to check framework flexibility.

Ensemble metrics and rendering

• Compute point metrics (RMSE, bias, R2) on the ensemble mean and spread metrics (coverage, CRPS) on the full ensemble.
• Add metric_CRPS on scoringRules::crps_sample and register it
• Draw the member spaghetti under the q05/q95 ribbon: reshape ensemble_output.csv with efi_long_to_array() keeping every member. Keep the reshape helper in the toolkit with a test; do not port it to PEcAn.utils for now. ( that's a deferred discussion on our list)

Data intake and tests

• Add a small testthat integration test, a subset of members and timesteps that verifies the pipeline runs and the scorecard columns are correct. Keep it fast and deterministic; do not run a full ensemble in CI.

Deliverables

• The AmeriFlux benchmark produces a Quarto scorecard from a real model ensemble against real observations (point metrics on the mean, coverage and CRPS on the ensemble), and the Salinas SOC run produces its HTML scorecard.
• metric_CRPS is implemented, registered, and tested.
• The synthetic integration test passes in CI.

Out of scope / deferred

• Porting efi_long_to_array into PEcAn.utils.
• The tutorial vignette and full roxygen documentation, which are Phase 5. The Salinas run can be written up as a walkthrough to seed that vignette.
• Any BETYdb coupled path.

Open dependencies

• The AmeriFlux MVP needs a real SIPNET ensemble at the benchmark site(s), produced upstream like the Salinas handoff. (@divine7022 generating model ensemble and hand off the EFI-long ensemble_output.csv )
• Make sure it's consistent with Yang Gu's SDA validation work

References

• Workplan: https://docs.google.com/document/d/1A41efiS7XX-2g8KkEKpg8nTFHoNeaWj15FbqauZ8fJA
• PRs: Refactor the baseline Validation Pipeline to align with GSoC Architecture #4017 (registry, align_by_time), Add testthat coverage for remaining numeric metrics  #4032 (metrics, PMU/coverage), feat: implement automated reporting and visualization layer #4040 (reporting and Quarto, Phase 3), Take over PR #3888: Add unit tests for benchmarking metrics and document metric_R2 #4041 (metric_R2)
• Model side and EFI: ccmmf/downscaling/R/efi_long_to_arrays.R