feat(upload): read shipboard CTD files into the core model (plan W1)

An uploaded cast can now be QC'd before it reaches a release. The design
principle that makes this cheap: every rule targets obs/sample, so projecting a
file into that shape runs the whole registry unchanged — no rule learns anything
about where the data came from.

FORMATS, and what each costs

  .csv  the CalCOFI cast file; maps through measurement_type.csv _source_column,
        which already holds exactly those names.
  .cnv  Sea-Bird converted. Its `# name N =` header names every column, so there
        is nothing to infer. PREFERRED.
  .asc  Sea-Bird ASCII. Same vocabulary, but see below.
  .btl  bottle summary: one Date header word over three data fields, several
        tagged statistic rows per bottle.
  .hex  REFUSED, with the reason. It is raw A/D counts; converting it needs the
        .xmlcon calibration file. A best-effort conversion would be invented
        numbers presented as measurements, which in a QC tool is the worst
        possible output.

THE TRAP IN .asc, measured not assumed

The header is fixed-width and adjacent names RUN TOGETHER —
`Sbeox0ML/LSbeox0Mm/Kg`. In 179 of 200 CalCOFI files a whitespace split gives the
wrong column count and silently mis-assigns every column after the collision.
Both names and numbers are right-aligned, so sbe_split_header() cuts the header at
the data rows' stop positions. When the result is not self-consistent it ERRORS
and asks for the .cnv rather than guessing. Measured across the archive: ~86% of
.asc and ~47% of .btl read cleanly; the rest say why.

MAPPING

Two vocabularies, one answer, both in reviewable registries rather than in code:
the CalCOFI names through measurement_type.csv, the Sea-Bird names through the new
metadata/sbe_name_map.csv. Unmapped columns are a RESULT, not an error — they are
where a format change announces itself. Raw voltages are unmapped deliberately
(which sensor a V0 is depends on the wiring), and FlECO-AFL is left unmapped
because the registry carries the fluorometer VOLTAGE, so mapping the converted
value would silently change units.

The projection applies the same two repairs the pipeline already knows — the -99 /
-9.99e-29 sentinels are deleted, and quality codes stored as "9.0" are stripped
textually (not via an integer cast, which would round an unexpected "9.5") —
because a new file is exactly where those arrive.

Also: qc_cast_profile's sibling qc_upload_con() makes the upload obs / sample /
obs_ctd_full in memory, so an uploaded cast (which IS full resolution) gets the
profile rules too rather than skipping them.

Verified end to end on a real .btl from the CalCOFI archive: all 16 rules ran,
5 correctly reporting skip because that bottle file carries no temperature_ave.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GEVWNQXLvWzz25oF8hqESd

Author: bbest

NAMESPACE

@@ -55,6 +55,8 @@ export(create_gcs_manifest)
 export(create_lookup_table)
 export(create_redefinition_files)
 export(csv_to_parquet)
+export(ctd_map_columns)
+export(ctd_upload_to_core)
 export(delete_flagged_rows)
 export(delete_gcs_prefix)
 export(derive_cruise_key_on_casts)
@@ -142,17 +144,23 @@ export(qc_run_all)
 export(qc_run_rule)
 export(qc_stage_reference)
 export(qc_summarize)
+export(qc_upload_con)
 export(query_at_time)
 export(read_calcofi_meta)
 export(read_csv_files)
 export(read_csv_metadata)
+export(read_ctd_upload)
 export(read_ingest_manifest)
 export(read_ingest_parquet)
 export(read_ingest_yaml)
 export(read_input_fingerprint)
 export(read_measurement_type)
 export(read_parquet_table)
 export(read_relationships_json)
+export(read_sbe_asc)
+export(read_sbe_btl)
+export(read_sbe_cnv)
+export(read_sbe_header)
 export(record_schema_version)
 export(register_measurement_types)
 export(release_excluded_datasets)
@@ -162,6 +170,7 @@ export(report_ship_matches)
 export(sample_arm_self)
 export(save_duckdb_to_gcs)
 export(save_working_ducklake)
+export(sbe_split_header)
 export(scan_metadata_gaps)
 export(set_duckdb_comments)
 export(show_fields_redefine)

NEWS.md

@@ -29,6 +29,37 @@ core-projection `switch()` arms produced.
 
 New `Suggests: terra` (only for `qc_stage_reference(gebco_tif = )`).
 
+## Uploads: shipboard files -> the core model
+
+An uploaded cast can now be checked before it ever reaches a release. The design
+principle that makes it cheap: every rule targets `obs` / `sample`, so projecting
+a file into that shape runs the whole registry unchanged.
+
+- **`read_ctd_upload()`** — dispatches on extension: `.csv` (CalCOFI cast file),
+  `.cnv`, `.asc`, `.btl`. `.hex` is **refused with its reason** — it is raw A/D
+  counts and needs the `.xmlcon` calibration file, so any conversion without it
+  would be invented numbers.
+- **`sbe_split_header()`** — the trap that makes `.asc` hard: the header is
+  fixed-width and adjacent names run together (`Sbeox0ML/LSbeox0Mm/Kg`) in **179
+  of 200** CalCOFI files, so a whitespace split mis-assigns every column after the
+  collision. Names and numbers are right-aligned, so columns are cut at the data
+  rows' stop positions — and when the result is not self-consistent it **errors
+  rather than guessing**, asking for the `.cnv` whose header is unambiguous.
+  Measured: ~86% of `.asc` and ~47% of `.btl` read cleanly; the rest say why.
+- **`read_sbe_cnv()` / `read_sbe_asc()` / `read_sbe_btl()` / `read_sbe_header()`**
+  — including `bad_flag` → `NA`, and the `.btl` quirks (one `Date` header word over
+  three data fields; several tagged statistic rows per bottle).
+- **`ctd_map_columns()`** — CalCOFI names map through `measurement_type.csv`
+  `_source_column`; Sea-Bird names through the new
+  `metadata/sbe_name_map.csv`. **Unmapped columns are a result, not an error** —
+  they are where a format change announces itself.
+- **`ctd_upload_to_core()`** — the projection, applying the same `-99` /
+  `-9.99e-29` sentinel deletion and `"9.0"` → `"9"` quality-code repair the
+  pipeline already knows, because a new file is exactly where those arrive.
+- **`qc_upload_con()`** — an in-memory connection where the upload *is*
+  `obs` / `sample` / `obs_ctd_full`. Nothing touches a release; it dies with the
+  session.
+
 ## Cast profiles for review
 
 - **`qc_cast_profile()`** — the full-resolution scans for the physical cast a