feat(check): the cruise that leaves obs and keeps its casts (3.12.0)

Release v2026.08.08 shipped 10 calcofi_ctd-cast cruises holding all 1,186 of
their casts and none of their 874,000 observations. Nothing in the pipeline
said a word, and nothing was going to: PK/FK validation runs child -> parent,
so every surviving obs row still had a parent cast, and a parent with NO
CHILDREN violates no constraint. The bounds backstop inspects obs, which these
cruises had entirely left. No check anywhere looked at the parent side.

check_cruise_coverage() is that check. Three things it has to get right:

- The grain is the CRUISE, not the sample. A CTD `sample` row is one physical
  cast per DIRECTION while obs keeps one direction, so ~half of that dataset's
  cast rows legitimately carry no observations; a per-sample assertion is wrong
  on arrival. A whole cruise with none never is.
- It joins through sample_key, never obs.cruise_key — that denormalized column
  is NULL on 59,274 swfsc_cufes rows and 14,170 euphausiid ones and would
  invent orphans that do not exist. A test drops the column outright so an
  implementation that reaches for it fails rather than passing quietly.
- A dataset emitting no observations at all is exempt. sio_pic-zooplankton is a
  net-tow registry whose biovolumes are pending from the provider: 587
  sample-only cruises that must not fail. The rule is relative, so no allowlist
  is needed to say so.

max_orphan_cruises takes a named per-dataset vector, so the release can ratchet
a documented backlog while a new orphan still fails; an ingest that knows its
own correct answer passes 0.

Also fixes a stale fixture in test-in_release.R that declared a directory
`output:` and had been failing since check_nested_outputs() landed in 3.11.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: bbest

DESCRIPTION

@@ -1,6 +1,6 @@
 Package: calcofi4db
 Title: CalCOFI Database Tools
-Version: 3.11.0
+Version: 3.12.0
 URL: https://calcofi.io/calcofi4db
 BugReports: https://github.com/calcofi/calcofi4db/issues
 Authors@R: 

NAMESPACE

@@ -29,6 +29,7 @@ export(build_taxon_table)
 export(cc_stage_dir)
 export(cc_stage_path)
 export(changed_inputs)
+export(check_cruise_coverage)
 export(check_data_integrity)
 export(check_measurement_bounds)
 export(check_multiple_datasets)

NEWS.md

@@ -1,3 +1,38 @@
+# calcofi4db 3.12.0
+
+## `check_cruise_coverage()` — the cruise that leaves `obs` and keeps its samples
+
+Release `v2026.08.08` shipped 10 `calcofi_ctd-cast` cruises that had lost **every
+one of their 874,000 observations** while keeping all 1,186 of their casts. The
+CTD transects app went from 142 cruises to 132 overnight; nothing in the pipeline
+said a word.
+
+Nothing was going to. PK/FK validation runs child -> parent, so every `obs` row
+that remained still had a parent cast — and a parent with **no children** violates
+no constraint. The bounds backstop only inspects `obs`, which these cruises had
+entirely left. There was no check anywhere that looked at the parent side.
+
+`check_cruise_coverage(con)` is that check: one row per `dataset_key` with
+`cruises`, `cruises_no_obs` and `orphan_samples`, halting when a dataset exceeds
+its allowance. Three things it gets right that a first cut would not:
+
+- **The grain is the cruise, not the sample.** A CTD `sample` row is one physical
+  cast *per direction* while `obs` keeps one direction, so ~half of
+  `calcofi_ctd-cast`'s cast rows legitimately carry no observations. A per-sample
+  assertion is wrong on arrival; a whole cruise with none never is.
+- **It joins through `sample_key`, never `obs.cruise_key`.** That denormalized
+  column is NULL on 59,274 `swfsc_cufes` rows and 14,170 euphausiid ones, which
+  would invent orphans that do not exist.
+- **A dataset emitting no observations at all is exempt.** `sio_pic-zooplankton`
+  is a net-tow registry whose biovolumes are pending from the provider, so
+  `sample`-only is its designed state — 587 cruises that must not fail. The rule
+  is relative ("if a dataset contributes observations, every one of its cruises
+  must"), so it needs no allowlist to say so.
+
+`max_orphan_cruises` takes a named per-dataset vector so the release can ratchet a
+documented backlog while a *new* orphan still fails; an ingest that knows its own
+correct answer passes `0`.
+
 # calcofi4db 3.11.0
 
 ## `build_targets_list()` refuses a directory `output:`

R/check.R

@@ -483,3 +483,104 @@ check_taxon_ids <- function(con, allow = character(), halt = TRUE,
 
   if (verbose) rpt else invisible(rpt)
 }
+# check_cruise_coverage --------------------------------------------------------
+
+#' Cruises that carry samples but no observations — the silent-loss guard
+#'
+#' A cruise can leave `obs` without leaving `sample`, and nothing about that
+#' violates a foreign key: FK validation runs child -> parent, so every surviving
+#' `obs` row still has a parent, and a parent with **no children** breaks no
+#' constraint. Release `v2026.08.08` shipped in exactly that state — 10
+#' `calcofi_ctd-cast` cruises kept all 1,186 of their casts and lost all 874,000
+#' of their observations, because a Google Drive placeholder read as zero rows and
+#' the direction letter the thinning step needs came from the filename of a
+#' conflict copy. No check anywhere looked at the parent side.
+#'
+#' The grain is the **cruise**, deliberately, and it is not the sample. A CTD
+#' `sample` row is one physical cast *per direction* while `obs` keeps a single
+#' direction, so about half of `calcofi_ctd-cast`'s cast rows legitimately carry
+#' no observations and a per-sample assertion would be wrong on arrival. A whole
+#' cruise with none is never legitimate.
+#'
+#' A dataset that emits **no** observations at all is exempt rather than failing
+#' 587 times: `sio_pic-zooplankton` is a net-tow registry whose biovolumes are
+#' still pending from the provider, so contributing `sample` alone is its designed
+#' state. The rule is therefore relative — *if a dataset contributes observations,
+#' every one of its cruises must* — which needs no allowlist to say so.
+#'
+#' @param con a DBI connection holding `sample` and `obs`
+#' @param obs_tbl name of the observation table (default `"obs"`)
+#' @param max_orphan_cruises integer allowance, or a named integer vector keyed by
+#'   `dataset_key` for a per-dataset ratchet. Use `0` where the correct answer is
+#'   known to be zero (an ingest asserting its own output); use the current counts
+#'   as a ratchet at release time so a *new* orphan fails while a documented
+#'   backlog does not. May only ever be lowered.
+#' @param halt logical; `stop()` when the allowance is exceeded (default `TRUE`)
+#' @param verbose logical; message the summary
+#' @return a data.frame, one row per `dataset_key`, with `cruises`,
+#'   `cruises_no_obs`, `orphan_samples` and `emits_obs` (invisibly when
+#'   `verbose = FALSE`)
+#' @export
+#' @concept check
+check_cruise_coverage <- function(con, obs_tbl = "obs",
+                                  max_orphan_cruises = 0L,
+                                  halt = TRUE, verbose = TRUE) {
+  present <- DBI::dbListTables(con)
+  if (!all(c("sample", obs_tbl) %in% present))
+    stop(glue::glue(
+      "check_cruise_coverage(): needs `sample` and `{obs_tbl}` in `con`."))
+
+  # join through sample_key, never through obs.cruise_key: the denormalized
+  # cruise_key on obs is NULL for 59,274 swfsc_cufes rows and 14,170 euphausiid
+  # ones, which would invent orphans that do not exist.
+  rpt <- DBI::dbGetQuery(con, glue::glue("
+    WITH s AS (
+      SELECT dataset_key, cruise_key, sample_key
+      FROM sample WHERE cruise_key IS NOT NULL),
+    o AS (SELECT DISTINCT sample_key FROM {obs_tbl}),
+    j AS (
+      SELECT s.dataset_key, s.cruise_key, COUNT(*) AS samples,
+             COUNT(*) FILTER (WHERE o.sample_key IS NOT NULL) AS samples_with_obs
+      FROM s LEFT JOIN o USING (sample_key) GROUP BY 1, 2)
+    SELECT dataset_key,
+           COUNT(*)                                             AS cruises,
+           COUNT(*) FILTER (WHERE samples_with_obs = 0)         AS cruises_no_obs,
+           COALESCE(SUM(samples) FILTER (WHERE samples_with_obs = 0), 0)
+                                                                AS orphan_samples
+    FROM j GROUP BY 1 ORDER BY 1"))
+  rpt$emits_obs <- rpt$cruises_no_obs < rpt$cruises
+  # a registry-only dataset has no observations to lose
+  rpt$cruises_no_obs[!rpt$emits_obs]  <- 0L
+  rpt$orphan_samples[!rpt$emits_obs]  <- 0L
+
+  allow <- if (is.null(names(max_orphan_cruises))) {
+    stats::setNames(rep(as.integer(max_orphan_cruises)[1], nrow(rpt)),
+                    rpt$dataset_key)
+  } else {
+    a <- stats::setNames(rep(0L, nrow(rpt)), rpt$dataset_key)
+    a[names(max_orphan_cruises)] <- as.integer(max_orphan_cruises)
+    a
+  }
+  over <- rpt[rpt$cruises_no_obs > allow[rpt$dataset_key], , drop = FALSE]
+
+  if (verbose)
+    message(glue::glue(
+      "cruise coverage: {sum(rpt$cruises_no_obs)} cruise(s) with samples and no ",
+      "{obs_tbl} across {sum(rpt$emits_obs)} observing dataset(s) ",
+      "({sum(!rpt$emits_obs)} registry-only, exempt)"))
+
+  if (nrow(over)) {
+    detail <- paste(sprintf(
+      "  %s: %d of %d cruise(s), %d orphan sample(s) — allowance %d",
+      over$dataset_key, over$cruises_no_obs, over$cruises,
+      over$orphan_samples, allow[over$dataset_key]), collapse = "\n")
+    msg <- paste0(
+      "cruise(s) carry samples but no ", obs_tbl, ":\n", detail,
+      "\n  The casts survive and every FK still resolves, so nothing else will\n",
+      "  report this. Find where the observations were dropped before releasing;\n",
+      "  raising the allowance to make the check pass republishes the loss.")
+    if (halt) stop(msg, call. = FALSE) else warning(msg, call. = FALSE)
+  }
+
+  if (verbose) rpt else invisible(rpt)
+}

man/check_cruise_coverage.Rd

@@ -0,0 +1,68 @@
+# A cruise can leave `obs` without leaving `sample`, and no foreign key notices:
+# FK validation runs child -> parent, so a parent with no children is silent.
+# These pin the four behaviors that make the check usable as a release gate.
+
+# two datasets: `aa` observes 2 of its 3 cruises, `rr` is a registry (no obs)
+make_con <- function(env = parent.frame()) {
+  con <- get_duckdb_con(":memory:")
+  withr::defer(DBI::dbDisconnect(con, shutdown = TRUE), envir = env)
+
+  smpl <- data.frame(
+    dataset_key = c(rep("aa", 5), rep("rr", 2)),
+    cruise_key  = c("c1", "c1", "c2", "c3", "c3", "r1", "r2"),
+    sample_key  = c("aa:1", "aa:2", "aa:3", "aa:4", "aa:5", "rr:1", "rr:2"))
+  # c3's two samples have no obs at all; c1 has one observed and one bare sample
+  # (the CTD up/down-cast case, which must NOT count as a loss)
+  obs <- data.frame(
+    sample_key       = c("aa:1", "aa:3", "aa:3"),
+    measurement_type = c("t", "t", "s"))
+
+  DBI::dbWriteTable(con, "sample", smpl)
+  DBI::dbWriteTable(con, "obs", obs)
+  con
+}
+
+test_that("a cruise with samples and no obs is found, and a half-observed one is not", {
+  con <- make_con()
+  rpt <- check_cruise_coverage(con, halt = FALSE, verbose = FALSE) |>
+    suppressWarnings()
+
+  aa <- rpt[rpt$dataset_key == "aa", ]
+  expect_equal(aa$cruises, 3L)
+  # c3 only — c1 keeps one bare sample beside an observed one and is fine
+  expect_equal(aa$cruises_no_obs, 1L)
+  expect_equal(aa$orphan_samples, 2)
+})
+
+test_that("a dataset that emits no observations at all is exempt, not 587 failures", {
+  con <- make_con()
+  rpt <- check_cruise_coverage(con, halt = FALSE, verbose = FALSE) |>
+    suppressWarnings()
+
+  rr <- rpt[rpt$dataset_key == "rr", ]
+  expect_false(rr$emits_obs)
+  expect_equal(rr$cruises_no_obs, 0L)   # sio_pic-zooplankton is a tow registry
+  expect_true(rpt$emits_obs[rpt$dataset_key == "aa"])
+})
+
+test_that("it halts by default and the per-dataset allowance ratchets it", {
+  con <- make_con()
+  expect_error(check_cruise_coverage(con, verbose = FALSE), "samples but no obs")
+  expect_silent(
+    check_cruise_coverage(con, max_orphan_cruises = c(aa = 1L), verbose = FALSE))
+  # the ratchet is per dataset: an allowance for another key does not cover `aa`
+  expect_error(
+    check_cruise_coverage(con, max_orphan_cruises = c(zz = 9L), verbose = FALSE),
+    "aa: 1 of 3")
+})
+
+test_that("a NULL cruise_key on obs cannot invent an orphan", {
+  # obs.cruise_key is denormalized and NULL for tens of thousands of real rows,
+  # so the join must go through sample_key. Drop the column entirely: if the
+  # implementation ever reaches for it, this fails rather than silently passing.
+  con <- make_con()
+  DBI::dbExecute(con, "ALTER TABLE obs ADD COLUMN cruise_key VARCHAR")
+  rpt <- check_cruise_coverage(con, halt = FALSE, verbose = FALSE) |>
+    suppressWarnings()
+  expect_equal(rpt$cruises_no_obs[rpt$dataset_key == "aa"], 1L)
+})

tests/testthat/test-check_cruise_coverage.R

@@ -116,7 +116,9 @@ test_that("a flagged-out ingest is not an [auto] dependency of the release", {
   writeLines(c(
     "---", "title: Release", "calcofi:", "  target_name: release_database",
     "  workflow_type: release", "  dependency:", "    - auto",
-    "  output: data/releases", "---"),
+    # a FILE, not `data/releases`: check_nested_outputs() (3.11.0) rejects a
+    # directory `output:`, and this fixture predates it
+    "  output: data/releases/_release_stamp.json", "---"),
     file.path(dir, "release_database.qmd"))
 
   tl  <- build_targets_list(dir, verbose = FALSE)