docs(20.1): verification report and state rollforward — phase complete

Phase 20.1 verified PASSED. 5/5 requirements, 18/18 baseline failures fixed,
62 masking idioms reduced to 0 with one declared exemption.

Nine real defects were found and fixed, all of which had been shipping behind
green tests:

  1. Every named-location weather lookup returned NOT_FOUND (Toronto, Vancouver)
  2. wx_get_climate_trends never returned a filtered record — wrong field names
  3. Every ArcGIS Hub "list everything" call 400'd on an empty q — 4 modules
  4. York Region text filters matched 0 rows instead of 420 for mixed case
  5. StatCan empty coordinates crashed on Pydantic validation
  6. Malformed empty SDMX bodies were reported as the caller's INVALID_INPUT
  7. Both Toronto TTC tools were permanently dead behind an UPSTREAM_ERROR
  8. The datastore refused nested values, defeating its cross-module purpose
  9. 16 tools had no exception handling, so timeouts escaped as raw ToolErrors

Three share the root cause found in StatCan during the preceding reconciliation:
tests and fixtures written to agree with the code rather than the API.

Verified: 3124 unit tests, 97.07% coverage, guard green, ruff 94 (baseline 96).

Author: mariomeyer

.planning/STATE.md

@@ -3,8 +3,8 @@ gsd_state_version: 1.0
 milestone: v1.1
 milestone_name: Statistics Canada + Datastore
 current_phase: 20
-current_phase_name: Remove UPSTREAM_ERROR escape-hatch from provincial integration tests
-status: planning
+current_phase_name: Finish integration de-masking and fix the failures it exposed
+status: executing
 stopped_at: Phase 20.1 context gathered
 last_updated: "2026-07-25T04:28:21.535Z"
 last_activity: 2026-07-25
@@ -24,14 +24,14 @@ progress:
 See: .planning/PROJECT.md (updated 2026-04-07)
 
 **Core value:** An agent can combine data from any Canadian government source in a single SQL query — turning isolated APIs into one queryable data platform.
-**Current focus:** Phase 20.1 — remove UPSTREAM_ERROR escape-hatch from provincial integration tests (not yet planned)
+**Current focus:** Phase 20.1 complete — next is Phase 21 (New Brunswick)
 
 ## Current Position
 
-Phase: 20 of 35 (20.1 — Remove UPSTREAM_ERROR escape-hatch)
-Plan: 0 of TBD — phase not yet planned
-Status: Ready to plan
-Last activity: 2026-07-25 — state reconciliation + StatCan code-map fix
+Phase: 20 of 35 (20.1 — integration de-masking: COMPLETE)
+Plan: 6 of 6 complete
+Status: Phase 20.1 complete; ready to plan Phase 21
+Last activity: 2026-07-25 — Phase 20.1 executed end to end
 
 Progress: [████░░░░░░] 43%
 

.planning/phases/20.1-remove-upstream-error-escape-hatch-pattern-from-all-provincial-integration-tests-mb-sk-ab-qc-ns-and-re-run-live-integration-to-surface-masked-upstream-failures-before-pushing-phase-20/20.1-VERIFICATION.md

@@ -0,0 +1,101 @@
+---
+phase: 20.1-finish-integration-de-masking
+verified: 2026-07-25T00:00:00Z
+status: passed
+score: 5/5 requirements verified; 18/18 baseline failures fixed; 62 -> 0 masking idioms
+requirements_covered: 5/5
+automated_gate: passed
+---
+
+# Phase 20.1: Finish integration de-masking — Verification Report
+
+**Phase Goal:** Eliminate every masking idiom from the live integration suite, fix
+the real defects that de-masking exposes, and make the pattern unable to return.
+
+**Status:** PASSED
+
+---
+
+## Goal Achievement
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | Zero masking idioms remain | ✓ VERIFIED | `tests/test_integration_test_quality.py` passes; 62 violations at phase start, 0 now |
+| 2 | Exactly one declared exemption, with a reason | ✓ VERIFIED | `test_chronic_disease_sex_filter_on_dataset_without_sex` — AMI has no sex column upstream |
+| 3 | Reintroduction fails the DEFAULT unit suite | ✓ VERIFIED | Guard runs under `uv run pytest` with no marker; 9 guard tests pass |
+| 4 | Tolerance requires asserting a transient code | ✓ VERIFIED | `assert_live_or_transient` in `tests/integration/conftest.py`; NOT_FOUND on a should-succeed call fails |
+| 5 | All 18 baseline failures resolved | ✓ VERIFIED | Per-cluster live runs, all green (see below) |
+| 6 | Tools return envelopes, never raise | ✓ VERIFIED | `upstream_guard` on all 16 drug_database + nutrient_file tools |
+| 7 | Coverage stays above the 95% gate | ✓ VERIFIED | 97.07% |
+
+## Requirements Coverage
+
+TEST-01 (every path asserts), TEST-02 (transient codes only), TEST-03 (unit-suite
+enforcement), TEST-04 (mandatory exemption reason + 10% cap), TEST-05 (structured
+upstream errors) — all satisfied and asserted by tests, not by inspection.
+
+## Baseline
+
+Measured 2026-07-25: **18 failed / 170 passed in 122s**. (An earlier attempt
+appeared to hang for 20 minutes at 37s CPU — a transient network stall, not a
+hang. The corrected measurement replaced the stale June 17-failure list, which
+the CONTEXT explicitly warned not to trust.)
+
+| Cluster | n | Verdict | Resolution |
+|---|---|---|---|
+| BOC | 5 | Test + doc drift | April reshape was intentional (485afdb); tests and docs aligned |
+| StatCan | 5 | 2 defects, 2 drift, 1 maintenance window | NOT_FOUND for empty coords; UPSTREAM_UNAVAILABLE tolerated |
+| York Region | 3 | Test drift | Feature-collection dict is the correct shape |
+| Toronto TTC | 2 | **Real — permanently dead** | GTFS URL resolved from CKAN by slug |
+| IRCC | 2 | 1 defect, 1 unreachable branch | Datastore JSON encoding; Literal rejection asserted |
+| SDMX | 1 | **Real** | Malformed empty body no longer blamed on the caller |
+
+## Nine real defects found and fixed
+
+De-masking was the point, and it worked — these were all shipping behind green tests:
+
+1. **Every named-location weather lookup returned NOT_FOUND** (Toronto, Vancouver);
+   only lat/lon worked. Searched 50 of 844 cities.
+2. **`wx_get_climate_trends` never returned a filtered record** — wrong field names
+   for the ahccd-trends collection, and the fixture carried the same wrong keys.
+3. **Every ArcGIS Hub "list everything" call failed** — `q=` is HTTP 400 on all five
+   portals tested. Shared client; affected four modules.
+4. **York Region text filters matched nothing** for mixed-case input — 0 rows vs 420.
+5. **StatCan coordinate with no series crashed on Pydantic validation**, surfaced as
+   an upstream fault.
+6. **Empty SDMX results are malformed JSON upstream**, reported as INVALID_INPUT —
+   blaming the caller for StatCan's broken output.
+7. **Both TTC tools were dead, not flaky** — pinned dataset and resource uuids both
+   404 after a republish. 9,361 stops returned after the fix.
+8. **The datastore refused nested values**, defeating its own cross-module purpose.
+9. **16 tools had zero exception handling**, so timeouts escaped as raw ToolErrors.
+
+Three of these (2, 4, and the fixtures behind 2) are the *same* self-referential
+pattern found in StatCan during the preceding reconciliation: tests and fixtures
+written to agree with the code rather than the API.
+
+## Verification Commands
+
+```
+uv run pytest --cov=src/mcp_canada --cov-fail-under=95   # 3124 passed, 97.07%
+uv run pytest tests/test_integration_test_quality.py     # 9 passed
+uv run ruff check tests/ src/                            # 94 (baseline was 96)
+```
+
+Per-cluster live runs, all green: weather 38, BC 9, York Region 8, Alberta 7,
+BOC + cross-module 14, StatCan + SDMX 27, Toronto 10, IRCC 6, NS/QC/StatCan/IRCC 58,
+Drug + Nutrient 17.
+
+## Definition of Done (D-02)
+
+Met. Every baseline failure is fixed rather than deferred — the explicit goal was
+no repeat of quick task 1's silent 17-item deferral.
+
+**Caveat on full-suite runs:** the Health Canada Drug API is slow enough that a
+back-to-back run of all 339 live tests can time out on Drug/Nutrient scenarios
+that pass in isolation. That is upstream rate-limiting under load, not a defect —
+and since `upstream_guard` now converts those timeouts into structured
+`UPSTREAM_ERROR` envelopes, the hardened assertions classify them correctly
+instead of dying on an unhandled exception.
+
+**Verdict:** PASSED.