diag: Two things happened, one causal and one cosmetic-but-real.

[norrietaylor]

Filed from diag-portal.

Bundle: minimal-diag-20260812T163943Z.tar.zst
Reporter context: Session was disconnected abruptly. Mouse events now written to terminal

---

Diagnosis

Verdict

Two things happened, one causal and one cosmetic-but-real. The session ended because the sandboxed process tree inside the guest was SIGKILLed (hakoniwa reported code=125 … container received signal SIGKILL), which closed every pty slave and surfaced as EIO on the pty master — minimald then tore the attachment down via the ProcessExited path. That path is the only teardown path in minimald that does not send the terminal "unwind codes" to the attached client, so the user's terminal was left with mouse reporting (and any other DEC private modes zellij had set) still enabled — hence mouse movements now printing escape sequences. The terminal corruption is a minimald teardown bug (crates/minimald/src/session_host.rs), not a client/terminal problem.

Confidence

High for the mouse-events root cause (log-confirmed teardown reason plus the exact code path that omits the reset). Medium-Low for why the sandbox got SIGKILLed — the bundle contains no killer attribution (no kernel OOM in the guest ring buffer, no killing session process log from minimald). Guest-side dmesg/audit at incident time, or a min client-side log for that connection, would raise it.

Skew check

• manifest.json .version (CLI that took the bundle): 0.5.0
• providers/local-minvmd0/socket-probe.json get_version: daemon 0.5.0, long_version 0.5.0, stdlib 0.0.18
• Guest providers/local-minvmd0/guest/daemon-diag/meta.json: 0.5.0, stdlib 0.0.18, pid 1, uptime 945097 s (≈10 d 22 h → started 2026‑07‑30 06:51)
• Host minvmd process 12010/12012 etime 13‑09:48 → started 2026‑07‑30 06:51; installed marker mtime 1785393923 = 2026‑07‑30 06:45:23 UTC → binaries installed 6 minutes before the daemons started, so nothing is running pre-install code.
• The only version disagreement in the bundle is historical: logs/minvmd.log.2026-07-28 lines carry service.version=0.5.0-rc3.dev.20.gf9de7a1b, from the pre-upgrade run that exited at 2026-07-30T06:45:17 ("VMM child exited"). The current run logs 0.5.0.

skew ruled out.

Evidence

• Guest daemon log, the incident (providers/local-minvmd0/guest/daemon-diag/logs/minimald.log.2026-08-12, mirrored at providers/local-minvmd0/boot.log:62203-62210):
  • 2026-08-12T16:37:08.402936Z WARN minimald::session_host pty master error; tearing down host {"error":"I/O error (os error 5)"}
  • 2026-08-12T16:37:08.413699Z WARN … DIAG hakoniwa container/process exited non-zero {"code":125,"exit_code":"None","reason":"container received signal SIGKILL"}
  • 2026-08-12T16:37:08.414508Z INFO … binding leaving mainloop {"reason":"ProcessExited"} (twice, .414 and .416 — both attached sessions died within 2 ms)
• Preceding normal activity: 16:32:57 session created … session_name="pkgs", 16:32:58 workspace upload complete bytes_received=16843267, 16:33:07 op::specs Building package: claude-code, 16:33:08.247 binding attached to session channel.
• The code path that explains the mouse breakage — crates/minimald/src/session_host.rs:479-505: BindingMsg::TeardownDueToProcessExit writes only an optional error string and breaks with ProcessExited, while TeardownDueToSuperceded/TeardownDueToDaemonShutdown/TeardownDueToDetach each do let _ = w.write_all(&unwind_codes).await; first. unwind_codes() (same file, ~line 2286) is exactly what disables "app keypad/cursor, paste, mouse" (clean.input_mode_diff(live)), leaves the alt screen (\x1b[?1049l), unhides the cursor, resets SGR and focus reporting. notify_remote_pty_err (line 2028) sends TeardownDueToProcessExit(Some(e)) — no unwind codes.
• No guest kernel OOM/kill: providers/local-minvmd0/guest/daemon-diag/logs/kmsg.txt ends at boot (EXT4-fs (vdb): mounted filesystem … r/w) — nothing was printed to the ring after boot, so the kernel did not kill anything at 16:37.
• Guest resources healthy: disk.json /var/lib/minimal 238 GB free; proc.txt init RSS 310 MB against 16 GiB booted RAM (status.json booted_ram_mib: 16384).
• Daemon never restarted and is healthy: status.json minvmd_alive: true, socket probe stat/connect/handshake/get_version all ok, uptime 10 d 22 h.
• User reattached after the drop: host process tree shows min proxy --socket …/ssh.sock with etime 00:45 (started ≈16:38:58), matching 16:38:59.146 binding attached to session channel in the guest log.
• Red herrings explicitly checked and dismissed: the CGNAT/Tailscale own-IP warnings (host/net/routes.txt, interfaces.txt) — the session transport is vsock over providers/local-minvmd0/ssh.sock, not the tailnet; host/power.txt sleep/wake is 2026‑08‑03, nine days before the incident; check's "missing errors.json" is a guest-bundle manifest nit (the digest shows the file present with 0 errors), unrelated.

Failure chain

1. 16:32:57 — min creates session pkgs (019ff6d1-…c4a92b), uploads 16.8 MB workspace, loadout dev composed (zellij, claude-code, vim, fzf, tailscale).
2. 16:33:07-16:33:08 — claude-code package build; shell attaches to the session channel; zellij (mouse reporting, alt screen) takes over the terminal.
3. 16:37:08.400 — pty master returns EIO: every slave fd closed, i.e. the sandbox process tree is gone. try_wait then reports hakoniwa code=125, reason=container received signal SIGKILL. Both attached session hosts hit this within 2 ms; no kernel OOM and no minimald-initiated kill is logged, so the killer is unattributed userspace action inside the guest.
4. 16:37:08.414/.416 — bindings leave the mainloop with reason=ProcessExited; the SSH channels are closed without writing unwind_codes(). The user sees an abrupt disconnect and their terminal keeps mouse tracking / DEC modes enabled → mouse movement is echoed as \x1b[<…M sequences.
5. 16:38:45-16:38:59 — user reconnects (min proxy restarts, ListSessions/GetSessionRecord, new binding attached at 16:38:59.146); both session records still status: active.
6. 16:39:43 — min bug collects this bundle (DiagBundleTarZst at 16:39:44.650).

Recommended next steps

• Immediate user workaround: printf '\e[?1000l\e[?1002l\e[?1003l\e[?1006l\e[?1049l\e[?25h\e[?1004l\e[m' or reset / stty sane in the affected terminal; re-attaching a healthy session also re-syncs the screen.
• Fix (minimal team): make the ProcessExited teardown carry the reset like the other three arms — change BindingMsg::TeardownDueToProcessExit(Option<io::Error>) to also carry self.unwind_codes() (computed in notify_remote_pty_err/notify_remote_process_exit, crates/minimald/src/session_host.rs:2028-2047) and write it before the error text / shell_exit_prompt (:479-489, :514). Note shell_exit_prompt itself renders an interactive dialog into a terminal still in the dead app's input modes — it needs the reset first regardless.
• Belt-and-braces: have the client (crates/minimal/src/attach.rs) restore terminal modes unconditionally on exit (RAII guard) so a daemon that never sends unwind codes — or an abrupt transport loss — cannot leave the tty in mouse mode.
• Separately investigate the SIGKILL: reproduce with guest dmesg -w captured, and add attribution to the hakoniwa exit path (log the signaller / cgroup memory.events / oom_kill counters) so reason=container received signal SIGKILL is not a dead end. Worth checking whether zellij/claude-code inside the sandbox self-terminated the process group, and whether the guest's per-session cgroup has a memory limit whose kills don't reach /dev/kmsg.

Caveats

• The bundle cannot show who sent SIGKILL: the guest kmsg ring contains only boot records, minimald logged no killing session process, and there is no host-side minimald/min client log for Aug 12 (logs/ holds only minvmd.log.2026-07-28/30; per logs/PROVENANCE.txt the daemon log lives in the guest bundle). A cgroup/oom kill that bypassed the console, an in-session kill -9, or a hakoniwa supervisor decision are all still possible.
• Both sessions dying in the same 2 ms suggests a single guest-wide trigger rather than one shell exiting; I could not identify it. If it recurs, that is the more important bug — the mouse artefact is a downstream cosmetic consequence.
• I did not find any evidence the abrupt drop was network-related (vsock/gvproxy/Tailscale) or version-related; if the user also saw RPC-level failures at the same time, that would need re-examination.