docs(rca): append 2026-05-26 cascade — health-checker loop, Coral

mismatch, EdgeTPU-only Frigate+ model

Frigate crash-looped a week after the SMS-spam RCA. Same #187 thread
(retired 192.168.1.x cameras), three more bugs underneath:

1. The skip-list in 20cb3f9 only suppressed the health-checker's
   majority_down check; the actual driver was api_unresponsive
   (kubectl exec failing during pod startup → rollout restart every
   5 min). Frigate itself was exiting because of repeated camera
   retry failures on the dead 192.168.1.x streams.
2. Disabling those 3 cameras at the Frigate config level (dc5d31b)
   exposed a 9-month-stale configmap pointing at the Coral USB TPU,
   which is no longer on pumped-piglet (confirmed via lsusb in VM
   105).
3. Switching to CPU detector exposed that the Frigate+ model is an
   EdgeTPU-only variant and won't run on CPU.

Also documents the cross-account git-push detour, the
CLAUDE.md-vs-reality drift on the detector, and the meta-lesson:
stale config is patient — none of these bugs were new, they'd been
carried for weeks-to-months and only surfaced once we changed the
timing by fixing the alert path on top.

Refs #187

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: kumar gopal

docs/blog/2026-05-19-frigate-sms-spam-two-watchdogs-one-skip-list-rca.md

@@ -101,4 +101,58 @@ The other piece of the same lesson: **don't have two systems that need to agree,
 
 ---
 
+## Update: 2026-05-26 — same #187 thread, three more bugs underneath
+
+A week after the SMS-spam fix, Frigate fell over with the user reporting "machine trying to do something in a loop." Same #187 chain (retired cameras), three new layers of failure exposed in sequence. Each fix surfaced the next bug. Five additional commits to get Frigate stable again.
+
+### What was actually happening
+
+The K8s health-checker was doing `kubectl rollout restart deployment/frigate` every 5 minutes because its `kubectl exec ... curl http://localhost:5000/api/stats` was failing with `container not found`. That classifies as `api_unresponsive` — a *different* code path from `majority_down`, which is what `FRIGATE_HC_SKIP_CAMERAS` (commit `20cb3f9`) suppresses. So the skip-list fix had no effect on this loop.
+
+But the health-checker wasn't the underlying cause. The Frigate container itself was exiting (clean code-0, `[INFO] Service Frigate exited with code 0 (by signal 0)`) every ~80 seconds. Walk the layers:
+
+1. **Frigate exited because the three retired cameras (`old_ip_camera`, `living_room`, `hall`) were still in `cameras:` in the configmap, just unreachable.** Frigate has been retrying their go2rtc streams for a month, hitting `404 Not Found` (camera not registered in go2rtc) and `i/o timeout` on `192.168.1.137`/`138`/`220`. Eventually some internal threshold trips and Frigate `sys.exit(0)`s itself to be restarted. The `FRIGATE_HC_SKIP_CAMERAS` change in `20cb3f9` only told the health-checker to ignore those cameras when computing `majority_down`; Frigate itself was never told. Issue #187 already named this open item — *"mark persistently-unreachable cameras as `enabled: false` in Frigate config rather than relying on health-checker skip list"* — and a week later it was still open. Fixed in `dc5d31b` (set `enabled: false` on all three).
+2. **With those cameras disabled, Frigate hit a *different* startup crash:** `[edgetpu_tfl] ERROR: No EdgeTPU was detected. Failed to load delegate from libedgetpu.so.1.0`. The Coral USB TPU is no longer plugged into pumped-piglet — `lsusb` inside VM 105 confirmed (only hub devices and the QEMU tablet). Per CLAUDE.md the Coral is on still-fawn now, idle. The Frigate configmap's `detectors:` block hadn't been updated since `ef2cfcf` (Feb 8) when Coral was passed through to pumped-piglet. Configmap was nine months stale. Fixed in `b0cfdc1` by reverting to the CPU detector (matches commit `926df02`, the proven pattern from the previous Coral-move).
+3. **With the CPU detector, Frigate failed config validation:** `Model does not support detector type of cpu`. The Frigate+ model `plus://c7b38453956cda87076baba4aca213e6` is an *EdgeTPU-only* variant — Frigate+ does not auto-deliver a CPU/ONNX-compatible format from the same URL when the model wasn't ordered for that target. Fixed in `9b7ce95` by commenting out the `model.path` entirely so Frigate uses its bundled default model.
+
+After `9b7ce95`, the pod came up `1/1 Running, 0 restarts`, both monitored cameras (`trendnet_ip_572w`, `reolink_doorbell`) recording at the expected 5-second cadence.
+
+### What got pushed (2026-05-26)
+
+| Commit | What |
+|---|---|
+| `e786e2c` | `suspend: true` on `frigate-health-checker` CronJob — same workaround as `5221d31` (Apr 18). |
+| `dc5d31b` | `enabled: false` on `old_ip_camera`, `living_room`, `hall` in `configmap.yaml`. |
+| `b0cfdc1` | `detectors: coral` → `detectors: cpu`. |
+| `9b7ce95` | Commented out `model.path: plus://...` so the CPU detector can boot with the bundled model. |
+
+Each commit was independently necessary; the next bug only became visible after the prior one was fixed. Total time, including the cross-account git-push detour described below, ~45 minutes once we were actually working the right problem.
+
+### CLAUDE.md was wrong
+
+CLAUDE.md states: *"Detector: ONNX on RTX 3070 (`nvidia.com/gpu: 1`) — Coral TPU no longer used by Frigate."* There has never been an ONNX detector in any commit on this repo. The configmap's `detectors:` block has only ever been `coral` (edgetpu) or `cpu`. CLAUDE.md was describing an aspirational target as if it were current state. Future-me reading this RCA: **don't trust CLAUDE.md as ground truth for current state — verify against the configmap / deployment / live cluster.**
+
+(Whether the eventual ONNX-on-RTX migration is even the right move is a separate question. The RTX 3070 is currently doing face-recognition embeddings; the embeddings code already complained at startup that OpenVINO couldn't find a GPU plugin and fell back to CPU. The GPU plumbing on this node has at least one gap.)
+
+### Cross-account git-push detour
+
+Three commits pushed clean. The next push 403'd: `Permission to homeiac/home.git denied to kumagopa_AglntEMU`. The `gh` CLI's active account is the work-domain one by default; only the personal `gshiva` account has write access to `homeiac/home`. Fix: `gh auth switch --user gshiva && gh auth setup-git && git push && gh auth switch --user kumagopa_AglntEMU`. `gh auth switch` is per-keyring-state and *did* survive across Bash calls in this harness for the duration of pushing, but the safe pattern is switch-push-switch-back as a tight sequence rather than leaving the personal account active. (Recorded to memory as `user_two_gh_accounts.md` so I don't re-learn this next session.)
+
+### New open items (in addition to those above)
+
+- [ ] **Fix the health-checker's circuit-breaker reset bug** so it can be un-suspended. The `last_restart_times` field in the `frigate-health-state` ConfigMap appears to clear between runs, so `MAX_RESTARTS_PER_HOUR` never fires. Already named in #187. Until this is fixed, the health-checker's value-add (paging when Frigate is actually broken) is below the cost (deployment churn when Frigate is taking >2 min to start). Currently suspended via `suspend: true` in `cronjob-health-python.yaml`.
+- [ ] **Decide on the Frigate+ model.** Either order an ONNX-format model for `c7b38453956cda87076baba4aca213e6` (Frigate+ provides this), or stay on the bundled default. Bundled default works fine for the two-camera load.
+- [ ] **ONNX-on-RTX detector setup, if that's the target state.** Plumb `detectors: onnx` (with TensorRT execution provider), verify the NVIDIA container runtime is delivering CUDA/TensorRT libraries to the pod, switch the model accordingly. Then update CLAUDE.md to match what actually exists.
+- [ ] **Reconcile CLAUDE.md with reality.** Specifically the lines describing the current Frigate detector and Coral location. Better: have CLAUDE.md describe the *intended* state under a "Target Architecture" heading and the *actual* state separately, so the drift is visible to future readers (human and agent).
+
+### The meta-lesson from the cascade
+
+Each layer of the fix was correct; none of them, alone, would have worked. The skip-list (`20cb3f9`) suppressed the obvious alert path but left the underlying camera-retry bomb ticking. Disabling the cameras (`dc5d31b`) defused that, exposing a Coral mismatch the configmap had been carrying for nine months. Switching to CPU (`b0cfdc1`) exposed that the Frigate+ model wasn't compatible. Dropping the model (`9b7ce95`) finally let Frigate boot.
+
+**Stale config is patient.** The Coral mismatch in the configmap was harmless for months because Frigate happened to keep starting before the retry-bomb triggered the exit. Once we changed the timing — by suppressing one of its symptoms — the next-most-fragile thing took over. None of these were *new* bugs; they were all things this repo had been carrying for weeks-to-months, invisible until they suddenly weren't.
+
+The user's framing during the incident — *"machine trying to do something in a loop"* — turned out to be true at three different levels: deployment loop, container loop, and process-internal retry loop. Each fix peeled one off.
+
+---
+
 *Related: [Out-of-Band SMS Notifications for Homelab Disaster Recovery](../source/md/blog-out-of-band-sms-disaster-recovery.md), [Same Lesson, Different Outage](2026-05-18-ups-plug-strikes-again-power-outage-rca.md), [Frigate health checker restart loop RCA](../runbooks/frigate-health-checker-restart-loop-rca.md) (#187)*