Skip to content

[ops] Teach the watchdog the on-demand era: tenant-aware, backup/unit-failure alerting, thermal/XID/VRAM telemetry #110

Description

@jphein

Severity: highfamiliar-watchdog was built for the always-on era and is provably blind in the new on-demand world: it reports green while the companion is intentionally down, probes HTTP only, and pages on intentional sleep while missing the failures that matter.

Fix — make the watchdog model reality:

  • Identify the :8080 tenant (companion vs qwen3-coder vs nothing) instead of assuming the companion is always up; don't page on intentional sleep/coder-mode.
  • Alert on the failures that actually bite: backup failure (20-byte artifact), failed/crashlooping units (EADDRINUSE), and DB disk pressure on /var/lib/mempalace-db (disk-full = the silent-write-failure class Slot picker: localhost variant URLs hit IPv6 ::1 trap; slots.json not seeded on deploy #86 keeps rediscovering).
  • Add GPU telemetry: thermal, XID/fell-off-bus events, and VRAM-headroom (the box runs an XID-79-history card at 81°C on one fan) — and run the benchmark doc's own deferred power-cap experiment.
  • Add the chat-stream-hang signal (see the no-timeout issue) so a wedged stream is detected, not silently held.

Raised by: drift, haze, cirrus.


From the 2026-06-09 dream-team premise audit (suggestions tier). Report: docs/audits/2026-06-09-premise-audit.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions