|
| 1 | +# Verification & QA |
| 2 | + |
| 3 | +How every feature gets verified before it ships. Layered, fast → thorough; **the running shell is |
| 4 | +the source of truth** and rigor scales to risk (a service that shells out and parses CLI output gets |
| 5 | +the full treatment; a static widget gets the fast path). Built from a research pass on QML/Quickshell/ |
| 6 | +Wayland test tooling (June 2026); see the tool table at the bottom. |
| 7 | + |
| 8 | +## The per-feature QA loop |
| 9 | + |
| 10 | +Apply top-to-bottom. Stop early for trivial widgets; run all of it for anything that parses output, |
| 11 | +shells out, handles IPC, or touches untrusted input. |
| 12 | + |
| 13 | +### 0. While developing — hot-reload |
| 14 | +Edit in a worktree (`~/dev/dotfiles-dev`), watch the live instance: |
| 15 | +```sh |
| 16 | +qs -c ii log -f # follow; or: qs -c ii log | tail |
| 17 | +``` |
| 18 | +⚠ **The running instance's log keeps STALE lines.** After a fix, don't trust the live buffer — confirm |
| 19 | +with a fresh isolated parse (step 2). This burned us once (the AndroidQuickToggleButton/HA spam). |
| 20 | + |
| 21 | +### 1. Static gate — `just lint` (now authoritative) |
| 22 | +`just lint` feeds qmllint the session `$QML_IMPORT_PATH` (`-I …`) so QtQuick/Quickshell types resolve |
| 23 | +(false positives drop ~16→2 per file) and filters the categories that stay structurally unresolvable |
| 24 | +on NixOS (the `qs.*` config imports + project singletons). What survives is real — it catches a typo'd |
| 25 | +property (`heigth`) that the old noisy lint buried. Also grep the injection antipattern: |
| 26 | +```sh |
| 27 | +grep -rnE '"bash".*"-c".*\+' --include='*.qml' . # value concatenated into a shell string = injection |
| 28 | +``` |
| 29 | + |
| 30 | +### 2. Load gate — full-shell parse (authoritative "did it load") |
| 31 | +There is no parse-only flag and QML errors do NOT exit the process, so the gate is: load the whole |
| 32 | +shell in an isolated instance and assert it reaches "Configuration Loaded" with no errors in your files. |
| 33 | +```sh |
| 34 | +qs -p ~/dev/dotfiles-dev/quickshell/ii > /tmp/parse.log 2>&1 & # then grep, then kill |
| 35 | +grep -iE 'Configuration Loaded' /tmp/parse.log # must appear |
| 36 | +grep -iE '<YourFile>|error|is not a type|cannot assign' /tmp/parse.log # must be empty |
| 37 | +``` |
| 38 | +Ignore the polkit / UPower / AiChat / KeyringStorage warnings — duplicate-instance artifacts. |
| 39 | + |
| 40 | +### 3. Service logic — deterministic parse tests (the bug-catcher) |
| 41 | +For anything parsing `nmcli`/`pactl`/`playerctl`/leases output: **test the parse against canned output**, |
| 42 | +not live hardware. Two ways: |
| 43 | +- **Fake-bin PATH shim** — drop a fake `nmcli` printing a known line into a temp dir on `$PATH`, run the |
| 44 | + service, assert. (Quickshell `Process` calls `execve` and inherits `$PATH`, so this intercepts even |
| 45 | + `bash -c "nmcli …"`.) |
| 46 | +- **Pure-function unit test** — split the parse into a plain JS function and run it under |
| 47 | + `qmltestrunner -platform offscreen` (or even `node`/`python` for a quick check). |
| 48 | + |
| 49 | +This is the layer that catches the substring-class bugs (`includes("activated")` matching |
| 50 | +`"deactivated"`; `includes("connected")` matching `"disconnected"`). When a feature depends on a tool's |
| 51 | +exact output, **run the tool and look** — don't assume (the `:activated` severity was over-rated because |
| 52 | +an agent assumed the nmcli output). |
| 53 | + |
| 54 | +### 4. IPC liveness — `qs ipc` |
| 55 | +```sh |
| 56 | +qs ipc --pid $(pgrep -f 'quickshell -c ii') call <target> status # service alive + sane state |
| 57 | +qs ipc show # list all IpcHandler targets |
| 58 | +``` |
| 59 | +Every service should expose a `status()` IpcHandler returning its key state. |
| 60 | + |
| 61 | +### 5. Interactive + visual (UI features) |
| 62 | +```sh |
| 63 | +just test-ui <feature> # ydotool drives the keybind, grim screenshots, Read the PNG |
| 64 | +``` |
| 65 | +Assert the layer surface actually exists with the right geometry (Hyprland IPC): |
| 66 | +```sh |
| 67 | +hyprctl -j layers | jq '.. | objects | select(.namespace? | startswith("quickshell:"))' |
| 68 | +hyprctl -j monitors | jq '.[] | select(.focused) | .reserved' # bar exclusive zone |
| 69 | +``` |
| 70 | +Optional **visual regression**: `grim -g "<geom>" actual.png` then `odiff golden.png actual.png diff.png |
| 71 | +--aa --threshold 0.05 --fail-on-layout` (the `--aa` skips antialiased font noise). |
| 72 | + |
| 73 | +### 6. Adversarial review (before ship) |
| 74 | +Run the multi-agent review on the diff (Sonnet, cost-controlled): |
| 75 | +the `qa-session-review` workflow — finders per dimension (bugs/vulns/races/redundancy) → each finding |
| 76 | +adversarially verified. It found the real hotspot bugs this layer is meant to catch. |
| 77 | + |
| 78 | +### 7. Security |
| 79 | +- Injection: the grep in step 1; **always pass tainted values via the Process `environment` map**, never |
| 80 | + interpolate into `bash -c`. |
| 81 | +- `shellcheck` on any helper `.sh`. |
| 82 | +- `gitleaks detect --no-banner` for secrets (stronger than a hand grep); keep committed files |
| 83 | + publication-safe (no hostnames/IPs/tokens — `example.com` placeholders). |
| 84 | +- Any externally-controlled string in a `StyledText` → `textFormat: Text.PlainText` (StyledText defaults |
| 85 | + to AutoText = HTML; a DHCP hostname could inject markup). |
| 86 | + |
| 87 | +## Tooling |
| 88 | + |
| 89 | +Already in the loop: `qmllint` (now `-I`-fed), `qs -p` / `qs ipc` / `qs log`, `just test` (IPC selftest), |
| 90 | +`just test-ui` (ydotool + grim), `hyprctl -j layers/monitors`, `dbus-monitor`. |
| 91 | + |
| 92 | +Worth adding (NixOS `home.packages` / a devShell): |
| 93 | + |
| 94 | +| Tool | Nix attr | Use | |
| 95 | +|---|---|---| |
| 96 | +| qmltestrunner, qmlformat, qmlls, qmlprofiler | `kdePackages.qtdeclarative` | unit-test pure QML logic offscreen; format; editor LSP; perf | |
| 97 | +| **GammaRay** | `gammaray` | attach to the running shell, inspect the live QML object tree / bindings / signals — best bug-hunting tool | |
| 98 | +| **shellcheck** | `shellcheck` | lint the helper `.sh` scripts (none today) | |
| 99 | +| **gitleaks** | `gitleaks` | secret scanning, pre-commit + CI | |
| 100 | +| **odiff** | `odiff` (npm `odiff-bin` if not yet in channel) | visual-regression diffing, AA-tolerant | |
| 101 | +| statix, deadnix | `statix`, `deadnix` | lint the NixOS flake (separate repo) | |
| 102 | + |
| 103 | +Input-injection note: `ydotool` (uinput, what we use) is compositor-agnostic and reliable; `wtype` is a |
| 104 | +zero-privilege wlroots alternative; `wlrctl` adds mouse + window-focus targeting for wlroots — handy if a |
| 105 | +test needs to focus a specific window before typing. |
| 106 | + |
| 107 | +## Headless CI (future) |
| 108 | +Fully-automated UI tests can run in a NixOS VM with Hyprland headless: `HYPRLAND_HEADLESS_ONLY=1`, QEMU |
| 109 | +`-vga none -device virtio-gpu-pci` (LLVMpipe software GL), then `hyprctl output create headless`, drive |
| 110 | +with ydotool, capture with grim, assert with `hyprctl -j` + odiff. Hyprland's own `nix/tests` is the |
| 111 | +reference recipe. Involved — current practice is the local loop above; this is the upgrade path. |
| 112 | + |
| 113 | +## Lessons baked in (hotspot QA cycle, 2026-06-30) |
| 114 | +- Verify a CLI's real output empirically before parsing it. |
| 115 | +- Field-exact parsing, never substring `includes()` on status strings. |
| 116 | +- Tainted values via `environment`, never shell-string interpolation. |
| 117 | +- `Text.PlainText` for any externally-controlled string. |
| 118 | +- The live log lies (stale lines) — confirm fixes with a fresh `qs -p`. |
0 commit comments