[v3.2.0] Implement TELSEM2 land atlas in CRTMv3

[BenjaminTJohnson]

Summary

Add the TELSEM2 (Tool to Estimate Land Surface Emissivities at Microwave frequencies) monthly-climatology atlas as a microwave land-surface emissivity source in CRTM. When a TELSEM2 coefficient file is loaded at CRTM_Init, it is used automatically for MW land emissivity; otherwise CRTM falls back to the existing NESDIS_LandEM model. This brings CRTM to parity with RTTOV, which uses TELSEM2 as its standard MW land emissivity atlas, and provides realistic, spatially-varying, monthly land emissivities for MW radiance assimilation and simulation.

https://journals.ametsoc.org/view/journals/atot/34/5/jtech-d-16-0188.1.xml

Motivation

CRTM's current MW land emissivity (src/SfcOptics/CRTM_MW_Land_SfcOptics.f90) uses the physical NESDIS_LandEM canopy/soil model below an 80 GHz cutoff and a constant 0.95 above it. TELSEM2 is an observation-derived (SSM/I) climatology that captures spatial and seasonal variability the physical model cannot, and is the de-facto standard in RTTOV. Adding it:

• improves MW land emissivity realism and RTTOV inter-comparability,
• gives users a well-validated alternative without removing the existing default,
• requires no new dynamic inputs (it is keyed only by latitude/longitude/month/frequency/angle).

Scope / design decisions

• Atlas packaging: convert the native RTTOV ASCII atlas to a CRTM-native coefficient file (netCDF, following the LSEatlas/SEcategory conventions), distributed via Get_CRTM_Binary_Files.sh + tarball md5sum.
• Algorithm fidelity: full port of RTTOV's mod_mwatlas (TELSEM2), not a simplified interpolation.
• Model selection: default-when-loaded — if a TELSEM2 atlas file is provided to CRTM_Init, it is used for MW land; otherwise NESDIS_LandEM is used. No new CRTM_Options flag (mirrors how CRTM_IRlandCoeff is used unconditionally when loaded).

Scientific scope (full TELSEM2 port)

Port the RTTOV mod_mwatlas algorithm:

• Equal-area grid (~0.25 deg, organized by latitude bands); compute the atlas cellnum from (lat, lon).
• 12 monthly atlases; each populated cell stores mean emissivity at 19, 37 and 85 GHz for V and H polarization, plus standard deviations and the inter-frequency/inter-polarization covariance used for interpolation.
• Optional spatial resolution control: average emissivity over all atlas cells within a requested radius (degrees) of the target
  point.
• Frequency interpolation/extrapolation to the channel frequency using the TELSEM2 covariance-based regression anchored at 19/37/85 GHz.
• Angular correction: apply the parameterized zenith-angle adjustment (separate V/H polynomial coefficients) to map atlas emissivities to the CRTM view angle.
• Polarization convention matches the existing MW land driver: SfcOptics%Emissivity(:,1)=V, (:,2)=H.
• No-data handling: TELSEM2 flags cells with no climatology (e.g., permanent ice/open water). On no-data, fall back to NESDIS_LandEM to preserve current behavior.

Data pipeline (atlas -> CRTM coefficient file)

• Build a one-time converter that reads the RTTOV ASCII atlas (ssmi_mean_emis_climato_MM_cov_interpol_M2 for MM = 01..12, plus the correlation file) and writes a CRTM-native netCDF coefficient file, following the LSEatlas/SEcategory IO conventions (Release/Version, global attributes, dimensions, Inquire/Read/Write).
• Define a TELSEM2Atlas_type capturing: grid geometry (per-band cell counts, first-cell indices), per-month emissivity/std/covarian
  ce arrays, frequency anchors, and angular-correction coefficients.
• Distribute the generated file via Get_CRTM_Binary_Files.sh and update the tarball md5sum (same mechanism as the IR atlas).

New source modules (mirroring the IR-land atlas pipeline)

New file	Template / analogue
src/Coefficients/EmisCoeff/MW_Land/TELSEM2/TELSEM2Atlas_Define.f90	src/Coefficients/EmisCoeff/IR_Land/LSEatlas/LSEatlas_Define.f90
src/Coefficients/EmisCoeff/MW_Land/TELSEM2/TELSEM2Atlas_netCDF_IO.f90 (+ optional _Binary_IO) and TELSEM2Atlas_IO.f90 wrapper	SEcategory_IO.f90 / SEcategory_netCDF_IO.f90
src/Coefficients/CRTM_MWlandCoeff.f90 (module-global MWlandC + CRTM_MWlandCoeff_Load/_Destroy/_IsLoaded)	src/Coefficients/CRTM_IRlandCoeff.f90
src/SfcOptics/MW_Land/TELSEM2/TELSEM2_Atlas_Module.f90 (cell lookup, spatial averaging, frequency interpolation, angular correction)	RTTOV mod_mwatlas

Integration points (edits to existing code)

• src/CRTM_LifeCycle.f90
  • Add an optional MWlandCoeff_File argument to CRTM_Init (mirror IRlandCoeff_File: default filename, Resolve_Coeff_Format,
    error handling).
  • Call CRTM_MWlandCoeff_Load(...) when a microwave sensor is present (ANY(SpcCoeff_IsMicrowaveSensor(SC))) and the file is provided.
  • Call CRTM_MWlandCoeff_Destroy(...) in CRTM_Destroy.
• src/SfcOptics/CRTM_MW_Land_SfcOptics.f90
  • Add a GeometryInfo argument to Compute_MW_Land_SfcOptics (lat/lon/month come from GeometryInfo%user%Latitude/Longitude/Month).
  • At entry: if CRTM_MWlandCoeff_IsLoaded() and the atlas returns valid data at (lat, lon, month, frequency, angle), use the TELSEM2 path and mark iVar so TL/AD return zero. Otherwise use the existing NESDIS_LandEM path (unchanged; retains its analytic Jacobians).
  • _TL/_AD: when the atlas path was taken, output zero sensitivities; the NESDIS_LandEM path is unchanged.
• src/SfcOptics/CRTM_SfcOptics.f90
  • Pass GeometryInfo into the forward Compute_MW_Land_SfcOptics call (currently at line ~518; GeometryInfo is already in scope, as it is passed to Compute_MW_Water_SfcOptics).
• src/CMakeLists.txt
  • Register all new .f90 files (explicit, non-glob source list).

Tangent-linear / adjoint

TELSEM2 depends only on latitude, longitude, month, frequency and view angle — none of which are CRTM state/control variables — so the TL and AD of the TELSEM2 path are identically zero. The TL/AD changes are minimal: detect the atlas path via iVar and zero the outputs (analogous to the existing high-frequency-default path). The existing NESDIS_LandEM analytic Jacobians (LAI, vegetation fraction, soil moisture; commit 9358caa) remain intact for the fallback path.

Validation / tests

• Coefficient file round-trip: Inquire/Read/Write consistency test for the new coefficient file.
• Science match vs RTTOV: compare CRTM TELSEM2 emissivities against RTTOV TELSEM2 for a set of (lat, lon, month, frequency, angle) points, V and H, to a tight tolerance (same algorithm and atlas).
• Forward integration: forward run on a MW sensor (e.g., ATMS / AMSU-A / SSMIS) with the atlas loaded vs NESDIS_LandEM; sanity-check brightness temperatures.

TL/AD consistency: extend the existing OMP / K-matrix consistency tests to confirm the TELSEM2-path TL/AD are zero and that the NESDIS_LandEM path is unaffected.

Risks / open questions

• Correctness of the equal-area grid cellnum indexing — port carefully and verify against RTTOV.
• Geometry latitude/longitude/month must be populated by the caller. If they are left at defaults, the atlas lookup may hit a no-data
  cell and fall back to NESDIS_LandEM; document this requirement.
• Behavior outside the 19-85 GHz anchor range (extrapolation vs capping) — match RTTOV and document.
• Coefficient-file format/versioning and binary-file distribution (md5sum, Get_CRTM_Binary_Files.sh) coordination.
• Thread-safety: the atlas is module-global, read-only after Init (consistent with other coefficients), so it is OpenMP-safe.

Acceptance criteria

• TELSEM2 atlas converted to a CRTM-native netCDF coefficient file and wired into Get_CRTM_Binary_Files.sh + md5sum.
• New TELSEM2Atlas_Define / IO and CRTM_MWlandCoeff modules implemented and building.
• CRTM_Init/CRTM_Destroy load/free the atlas via a new MWlandCoeff_File argument.
• Compute_MW_Land_SfcOptics uses TELSEM2 when loaded (with NESDIS_LandEM fallback on no-data) and is fed GeometryInfo.
• TL/AD return zero on the atlas path; NESDIS_LandEM Jacobians unchanged on the fallback path.
• (optional) Emissivities match RTTOV TELSEM2 within tolerance; forward/TL/AD/K regression and OMP-consistency tests pass.

[BenjaminTJohnson]

Status update — TELSEM2 implemented, integrated, and validated

Branch: feature/btj_REL-3.2.0. Core implementation is complete and building; the science is validated bit-for-bit against the RTTOV TELSEM2 reference, and the full CRTM test suite is green (201/201).

Delivered

• New modules
  • TELSEM2Atlas_Define — CRTM-native atlas type (all 12 months) + lifecycle
  • TELSEM2Atlas_netCDF_IO — Inquire/Read/Write the coeff file
  • TELSEM2_Atlas_Module — faithful port of RTTOV mod_mwatlas_m2 (equal-area grid, angular regression, >85 GHz frequency interpolation)
  • CRTM_MWlandCoeff — shared-data loader (Load/Destroy/IsLoaded)
  • Convert_TELSEM2_Atlas — one-time ASCII→netCDF converter tool
• Integration
  • GeometryInfo threaded into the MW land SfcOptics driver
  • CRTM_Init/CRTM_Destroy: new MWlandCoeff_File arg; drop-in load when TELSEM2.MWland.EmisCoeff.nc is present at the coeff path (mirrors PARMIO)
  • Compute_MW_Land_SfcOptics: uses TELSEM2 when loaded; falls back to
    NESDIS_LandEM on no-data cells (ocean / permanent ice)
• Test: test_TELSEM2_MWland (registered with CTest)
• Commits: 748e0f2 (feature), b561e03 (test)

Scope notes

• This is the real TELSEM2 (v2 / mod_mwatlas_m2): two surface classes per cell
  (class1 drives the angular correction, class2 the >85 GHz behaviour out to
  ~190 GHz for water-like classes).
• The atlas uncertainty (per-cell std + correlations) is not stored —
  it only feeds RTTOV's optional std/covariance outputs, which the CRTM surface
  optics do not consume. It can be added later as a Release/Version bump if a
  DA (observation-error) use arises.
• All 12 months live in one coeff file, since CRTM_Init loads coefficients once
  while the month is supplied per profile via Geometry.

Validation (all run)

• Bit-for-bit vs RTTOV test_telsem2 (month 9, lat −30, lon 302, θ=15°):
  30 GHz eV/eH = 0.954829 / 0.954446; resol 0.8° and the multi-freq case match.
• netCDF round-trip reproduces the golden values.
• End-to-end CRTM_Forward (amsua_n19, pure land): TELSEM2 ≠ NESDIS
  (ch1 23.8 GHz: 0.936 → 0.958), all channels physical; >85 GHz channels show the
  v2 interpolation rather than the flat 0.95 default.
• K-matrix and tangent-linear surface Jacobians (LAI / Vegetation_Fraction /
  Soil_Moisture_Content) are exactly 0.0 on the atlas path; the NESDIS path
  and its analytic Jacobians are unchanged.
• Full ctest suite: 201/201 pass (no regressions).

Acceptance criteria

• RTTOV ASCII atlas → CRTM-native netCDF coefficient file (+ converter tool)
• Atlas type, netCDF I/O, and shared-data loader modules implemented
• CRTM_Init/CRTM_Destroy wiring via MWlandCoeff_File (drop-in when present)
• Compute_MW_Land_SfcOptics uses TELSEM2 with NESDIS fallback; GeometryInfo threaded
• TL/AD identically zero on the atlas path; NESDIS Jacobians intact
• Emissivities match RTTOV TELSEM2 bit-for-bit
• Forward + K/TL + OMP regression suite passes
• Coefficient file published in the distributed fix/ tarball (checksum bumped; file staged)

[BenjaminRuston]

this is awesome @BenjaminTJohnson so I'll need some additional fix files or just a new tarball which assume can get via a branch (change path to tarball?)

@xian22 and @fcvdb we should test this it will likely have a very noticeable impact for MW sensors over land

[BenjaminTJohnson]

Some plots using ATMS_NPP OBS / GeoVals:

• Subset: all 137,525 snow-free land points (land_area_fraction>0.99 & snow<0.05) from the full 761k-obs NPP cycle → atms_npp_obs_landfull.h5 / atms_npp_geoval_landfull.nc4 (89 MB / 475 MB).
• Two hofx runs (run_{default,telsem2}_full.yaml), both clean (exit 0), ~14.5 GB peak RSS, well within budget.
• Figures: channels_full/ch01..22.png + channels_full.pdf (11 MB) — same 5-map + histogram layout, now with dense global land coverage.

Result holds at scale

The 137k-point statistics match the 6,000-point sample almost exactly, so the improvement is robust, not sampling noise:

• ch1 (23.8 GHz): bias −5.86 → +0.41 K, RMSE 10.81 → 7.01 K (ΔRMSE −3.80)
• ch2 (31.4 GHz): bias −7.55 → +0.45 K, RMSE 12.96 → 8.14 K (ΔRMSE −4.82)
• ch3 −1.49, ch16 −1.35, ch17 −0.07; sounding channels 8–15 untouched (ΔRMSE ≈ 0); 183 GHz negligible.

TELSEM2 removes NESDIS's systematic 5–7 K cold-obs window-channel bias over land and tightens RMSE, consistently across all continents.

Scope notes

• I kept it to land-covered, snow-free obs deliberately: TELSEM2 only drives the land-covered fraction in CRTM, so ocean/ice/snow obs would be identical between the two runs — running them would add ~5× compute for no signal.
• This is still a single 6-hour cycle (one NPP overpass set), so coverage is global in extent but one snapshot in time. Raw Obs − H(x), no bias correction.

[BenjaminRuston]

yes see a pretty massive improvement in the 89 GHz @BenjaminTJohnson

we should see if we can't get @xian22 to test this, and hopefully just make it standard practice

[BenjaminTJohnson]

FYI: UFO has no specific TELSEM2 or PARMIO switch: The selection happens entirely inside CRTM_Init, driven by whether the atlas file is present in the coefficient path that UFO hands it. UFO is oblivious to which land model runs.

Here's the full chain:

1. UFO passes the coefficient path, nothing more.
   The YAML obs options: CoefficientPath: is read into conf%COEFFICIENT_PATH (ufo_crtm_utils_mod.F90:578-579) and passed straight through as File_Path in the CRTM_Init call (ufo_radiancecrtm_mod.F90:341). That call forwards the standard coefficient args only — it does not pass MWlandCoeff_File (and UFO contains zero references to TELSEM2/MWland anywhere, it trusts CRTM to "do the right thing"). So UFO never names TELSEM2 specifically.

2. CRTM_Init auto-detects the atlas by file presence ("drop-in when present").
   Because no MWlandCoeff_File was passed, CRTM treats it as non-explicit and looks for the default name in the path (CRTM_LifeCycle.f90:1152-1163):

Resolved_MWlandCoeff_File = File_Path // 'TELSEM2.MWland.EmisCoeff.nc'
INQUIRE(FILE=..., EXIST=mwland_present)
IF (mwland_present) THEN
   CRTM_MWlandCoeff_Load(...)   ! loads atlas, sets module SAVE var MWlandC
END IF

• File present → atlas loads, CRTM_MWlandCoeff_IsLoaded() becomes .TRUE. (process-global state).
• File absent → nothing loads, IsLoaded() stays .FALSE. → NESDIS default gets loaded.

3. The runtime gate is per-profile in SfcOptics.
   CRTM_MW_Land_SfcOptics.f90:234: IF (CRTM_MWlandCoeff_IsLoaded()) → use TELSEM2_Emissivity at that lat/lon/month; if the atlas has no data there (or isn't loaded), fall through to NESDIS_LandEM.

So in this experiment (above) the only difference between the two runs is one YAML line — CoefficientPath:

• TELSEM2 run → …/coeff_telsem2/ (atlas symlinked in) → loaded → TELSEM2.
• Default run → stock …/build/ufo/test/Data/ (no atlas) → NESDIS.

Two consequences worth knowing:

• Because the detection is implicit, a missing atlas is a silent fallback to NESDIS (no error). That's why the default run "just works." (If a caller did pass MWlandCoeff_File explicitly, a missing file becomes a hard error — but UFO never does that.)
• It's process-global: once loaded, every land obs in that run uses TELSEM2. There's no per-obs or per-sensor selection.

This is also the exact same drop-in scheme CRTM uses for the PARMIO MW-water LUT (PARMIO.MWwater.EmisCoeff.nc), right above it in CRTM_LifeCycle. If you'd prefer an explicit YAML toggle (e.g. MWlandCoeff_File: or a Use_TELSEM2: flag) instead of file-presence, that would be a small UFO addition — forwarding the key into the CRTM_Init call — which doesn't exist today.

here's the relevant UFO commit message that enables the UFO capability:

commit 388187d4e287b881781b6911e749365d345b6e9c (HEAD -> feature/btj_REL-3.2.0)
Author: Benjamin T. Johnson <Benjamin.T.Johnson@noaa.gov>
Date:   Mon Jun 8 10:53:35 2026 -0400

    CRTM operator: set geometry lat/lon/date for location-aware surface models

    Load_Geom_Data populated only the view-geometry angles, leaving the CRTM
    geometry Latitude/Longitude at 0 and Year/Month/Day at their defaults
    (2001-01-01). The NESDIS surface models take all surface state from the
    GeoVaLs and are unaffected, so this was latent. The TELSEM2 MW land
    emissivity atlas, however, is a per-location, per-month climatology that
    reads lat/lon/month from the CRTM geometry: every profile resolved to
    (0N, 0E, January) -> open ocean -> no atlas data -> silent fall-back to
    NESDIS_LandEM, so the atlas never took effect through UFO.

    Populate geo% Latitude/Longitude (wrapped to 0-360 degE as CRTM requires)
    and Year/Month/Day from MetaData/dateTime. With this, TELSEM2 activates
    over land and the per-channel TB changes match expectations (large at MW
    window channels, zero in the 57 GHz O2 sounding band).