Publish METS as CF NetCDF: mets.nc (thinned) + mets_full.nc (full ~1-min)

[bbest]

Goal

Publish METS (underway TSG + meteorology) as CF NetCDF, in two files mirroring what ctd-cast already does:

• mets.nc — the thinned record (mets_thin), the headline product
• mets_full.nc — the full ~1-minute companion (mets_measurement, supplemental)

Reference implementation

publish_ctd-cast_to-netcdf.qmd (439 lines) is the template — same two-file split, same publish tail. Sections that map directly:

CTD section	notes for METS
## Setup	same; target_name must be underscored (publish_calcofi_mets_to_netcdf)
## Shape and integrity	adapt row/level counts
## Write the CF profile file	this is where METS differs — see below
## Verify	same shape: assert featureType, cf_role, counts, rowSize sums
## Full-resolution companion	reusable almost verbatim, including the chunking rationale
## Publish	cc_netcdf_plan() / cc_netcdf_manifest() — unchanged

The one real design difference

CTD is featureType = Profile — a set of casts, each a vertical stack of depth levels, encoded with rowSize ragged-array indexing and cf_role = profile_id.

METS is not a profile. It is a continuous track: one instrument, moving, sampling through time at ~1-minute resolution with no depth dimension at all. The right CF encoding is almost certainly featureType = trajectory with cf_role = trajectory_id on cruise_key, and the ragged-array rowSize counting samples per cruise rather than levels per cast. That substitution is the bulk of the work; the ragged-array machinery itself carries over.

Do not just swap the string — CF trajectory has its own required/recommended attributes, and the ACDD geospatial/time bounds are computed differently for a track than for a station set.

METS facts you'll need (all verified from the last render)

mets_sample	2,375,704 rows (after removing 5,431 duplicate timestamps)
mets_measurement	20,573,250 rows, 54 measurement types, partitioned by cruise_key, supplemental
mets_thin	511,953 rows (2.5%), 17 canonical types, 77,795 retained samples
core obs	471,376 rows at the underway grain, fed by mets_thin
coverage	2004-01 to 2022-11, 52 cruises with retrievable data

mets_thin carries a retained_reason column (grid = hourly baseline, inflection = RDP kept it because conditions deviated). That should survive into the NetCDF as a variable — a consumer needs to know a thinned track is non-uniformly sampled, otherwise they'll interpolate across a deliberately-preserved front as if it were evenly spaced.

Scale — probably simpler than CTD

The CTD companion needed per-partition chunked writes because obs_ctd_full is ~216M rows and OOM'd ERDDAP at every container size (see bench_erddap_ctd.qmd). METS full is 20.5M rows, ~10× smaller. It may fit a single pivot. Measure before importing the chunking complexity — but the partition-enumeration part is still needed:

read_parquet('.../**/*.parquet') 404s over HTTPS — expanding a glob needs a directory listing and object storage has none. Enumerate partitions via cc_release_partitions() and pass an explicit URL vector.

mets_measurement is partitioned by cruise_key, so that applies verbatim.

Known caveats to carry into the file's global attributes

1. Timezone is a team assumption, not provider-confirmed. The xlsx era (22 cruises, 2016-2022) has its DateTime treated as fixed -8:00 PST — see questions.csv mets_02. Every timestamp in those cruises inherits that assumption. A NetCDF asserting UTC without saying so would launder an open question into apparent fact.
2. Depth is recorded as surface (0 m), which is a placeholder — the hull seawater intake sits a few metres down and the offset likely varies by vessel (mets_25, deliberately left open). Relevant to anyone comparing underway SST/salinity against CTD or bottle values.
3. Two source families are missing entirely — every *_SCIMS.txt and *_SCS.txt link (2006-2008, 9 cruises each) returns 403 Forbidden. Coverage is 52 of 68 linked cruises, and the file should not imply completeness.
4. 1004MF had a broken published header (11 names for 12 fields, Longitude_W missing) repaired at ingest. Fixed in our pipeline; the file on calcofi.org is still wrong (mets_24).

Gotchas that will bite

• Set QUARTO_CHROMIUM_HEADLESS_MODE="new" and render via quarto::quarto_render(), not the quarto CLI.
• Headless Chrome does not exit after writing mermaid PNGs. Quarto then blocks forever on teardown even though the render is complete. Diagnostic: if _output/<nb>_files/figure-html/*.png and the HTML both exist, it's done — kill Chrome only and quarto exits cleanly. If no figures were emitted, that's the other failure: the mermaid graph is too large (always pass tables= to cc_erd()).
• Any cc_erd() call must pass tables=, or it diagrams every table in the connection and wedges Chrome permanently.

Definition of done

• mets.nc + mets_full.nc written and CF-verified
• retained_reason preserved in the thinned file
• The four caveats above stated in global attributes
• Notebook rendered and committed with its HTML (the whole point — unrendered publish code is what Ingest euphausiids, picoplankton-bacteria, mesopelagic-fish, METS #72 was about)
• Appears in the workflows index (scripts/build_workflows_index.R)

Why this is separate from #72

_full.nc reads the published parquet, so it touches nothing in #72 — that PR is verified and mergeable as-is. This was split out rather than half-built, because shipping an unrendered publish notebook is the exact defect #72 was opened to fix.