Add system tests for TUI flows across Linux, macOS, and Windows

[reacher-z]

Gap

The TUI (src/clawbench/tui.py, 1684 lines, 33 top-level functions) is currently untested end-to-end. Every mode branch and every platform-specific diagnostic path relies on manual smoke-testing.

Concretely, the following TUI entry modes exist and need coverage:

• mode_single (tui.py:425) — pick one model + one case
• mode_batch (tui.py:491) — matrix run, with _parse_range_input + _recommend_concurrent
• mode_human (tui.py:608) — noVNC human-drive flow
• mode_configure (tui.py:640) — models.yaml + secrets edit
• _add_model / _edit_model / _delete_model — model CRUD via questionary prompts
• Engine-diagnostic follow-up branches at tui.py:1101 (Darwin), tui.py:1121 (Linux), tui.py:1171 (Windows), tui.py:1187-1188 (is_mac / is_win), tui.py:1310 (podman VM), tui.py:1354 (Linux VM troubleshoot), tui.py:1431 (Docker daemon)

CI today runs on ubuntu-latest + macos-14 (see .github/workflows/ci.yml:24) and covers only unit tests in tests/. Windows is not in the matrix at all. Each of the platform.system() == "Windows" branches is currently dead-letter code from CI's perspective.

Proposed scope

1. TUI-level system tests (headless, scripted input)

Drive the TUI with scripted questionary / stdin input so each mode branch reaches its first real side-effect (subprocess spawn / file write) with mocks at that boundary. One happy-path test per mode:

• test_tui_single_happy_path
• test_tui_batch_range_input
• test_tui_human_mode
• test_tui_configure_add_model
• test_tui_configure_edit_model
• test_tui_configure_delete_model

Plus one engine-diagnostic branch test per platform fork:

• test_tui_engine_diagnostic_darwin_path
• test_tui_engine_diagnostic_linux_path
• test_tui_engine_diagnostic_windows_path

Mock platform.system() and _check_engine() so these run on any host.

2. Windows added to the CI matrix

Extend .github/workflows/ci.yml:

matrix:
  os: [ubuntu-latest, macos-14, windows-latest]

With fail-fast: false already in place, Windows failures don't block other platforms — but they surface as a regression signal. Related: #59 (os.sysconf AttributeError on Windows batch mode — this CI change would have caught that pre-merge).

3. Smoke-install tests on all three platforms

One job per OS that:

1. pip install dist/*.whl in a fresh venv
2. clawbench version prints non-empty
3. clawbench --help exits 0
4. clawbench cases lists ≥150 cases (guards against the symlink-packaging fragility in Harden bundled data against symlink-unaware build tools #67)
5. clawbench doctor runs and exits with 0 or 1 (not crash)

No container engine needed for these — they validate the wheel + CLI entry points only.

---

Supplementary details

Test infrastructure sketch

Existing tests live under tests/ (test_cli.py, test_engine.py, test_image.py, test_paths.py) — add a sibling tests/test_tui.py and a shared tests/conftest.py. Minimal fixture shape:

# conftest.py
import pytest, questionary

class _Canned:
    def __init__(self, answers): self._answers = iter(answers)
    def ask(self): return next(self._answers)

@pytest.fixture
def scripted_questionary(monkeypatch):
    """Replace questionary.{select,checkbox,text,confirm} with canned answers."""
    def _apply(answers):
        it = iter(answers)
        for name in ("select", "checkbox", "text", "confirm"):
            monkeypatch.setattr(questionary, name, lambda *a, **kw: _Canned([next(it)]))
    return _apply

@pytest.fixture
def fake_platform(monkeypatch):
    import platform
    def _apply(system): monkeypatch.setattr(platform, "system", lambda: system)
    return _apply

Each test then drives a single mode_*() call through scripted answers and asserts on the final side-effect (mocked run_cmd / subprocess.run).

Testability hooks that may need to land first

A few small refactors to tui.py make this tractable without string-parsing captured stdout:

• Factor the last-mile subprocess spawn in mode_single / mode_batch / mode_human behind a _launch(cmd: list[str]) seam so tests can replace it with a list-collector.
• Expose the assembled-command dict (summary passed to _confirm_launch) as the return value, so tests can assert on it without reading the confirm prompt.
• Add CLAWBENCH_NONINTERACTIVE=1 short-circuit in _confirm_launch that auto-returns True — lets smoke tests in CI exercise the mode all the way through without a TTY.

These are ~30 LOC total and should be in the same PR as the first tests.

Acceptance criteria

• tests/test_tui.py exists with the 9 tests above, all green on ubuntu / macos / windows runners
• .github/workflows/ci.yml matrix includes windows-latest
• A dedicated smoke-install job runs per OS and exercises version / --help / cases / doctor
• CLAWBENCH_NONINTERACTIVE=1 env honored; documented in a short paragraph in README.md "Development" section
• No new test depends on a running container engine (podman/docker) — that's follow-up work

Out of scope (deliberate)

• End-to-end container-run smoke tests on Windows — blocked by WSL2 provisioning in GH Actions, track separately.
• Visual-regression / screenshot diff of the TUI — questionary output varies by terminal emulator, not worth the flake cost.
• Migrating the test harness to pytest-textual-snapshot or similar — reserve for when/if we switch the TUI to Textual.

Why it's worth filing separately from #56 / #57

#56 is broad CI/CD rework and #57 is general refactor. This one is scoped specifically to: TUI mode branches × 3 operating systems, which is the highest-leverage regression surface (every user hits the TUI on first run via bare clawbench).

Priority

Medium-high. The TUI is literally the default command (clawbench with no args → TUI), and each platform fork is currently unverified by CI. Windows users have already hit #59 — we should catch the next one automatically.

[Perry2004]

Test cases added in #174.

[Perry2004]

CI done in #175