Skip to content

Commit 8b7c0e1

Browse files
coreytcoreyt
committed
docs: record TUI test methods and outcome
1 parent 2a908c6 commit 8b7c0e1

7 files changed

Lines changed: 145 additions & 17 deletions

‎dev/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ configuration, and operations.
1414
| Runtime boundaries and design rationale | [Architecture](architecture.md) |
1515
| Approved semantic input-injection design | [Prompt-injection classifier design](notes/design-prompt-injection-classifier.md) |
1616
| Accepted refactoring direction | [Refactoring analysis](notes/refactoring-analysis-2026-07-31.md) |
17+
| TUI test-maintenance methods and measured 0.5.15 outcome | [TUI test methods](notes/tui-test-methods-0.5.15.md) |
1718
| Latest published release | [0.5.12 plan](plans/0.5.12-plan.md), [PII egress canary](plans/runs/0.5.12-pii-egress-canary-2026-08-11.md), and [CHANGELOG](../CHANGELOG.md) |
1819
| Next engineering backlog | [0.5.14 TODO](plans/0.5.14-todo.md) |
1920
| OOM investigation instrumentation | [High-water runbook](debugging/instrumentation/oom-high-water.md) |

‎dev/notes/tui-test-methods-0.5.15.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# TUI test methods and outcome — 0.5.15
2+
3+
**Status:** accepted engineering guidance. This note covers test maintenance,
4+
not Airlock runtime or operator performance.
5+
6+
## Question and boundary
7+
8+
Slice 94 characterized Textual TUI tests, and Slice 95 tested one narrow
9+
change: whether eight pure Phase4 composition/navigation tests should use the
10+
existing explicit `AirlockApp(test_harness=True)` mode. This is not a claim
11+
about end-user TUI speed, proxy throughput, or the whole test suite.
12+
13+
Historical CI wall times are deliberately excluded from before/after results.
14+
Successful runs did not retain per-test JUnit timing and ran in different CI
15+
contexts, so they cannot establish a TUI cause.
16+
17+
## Test-mode policy
18+
19+
Textual's [testing guide](https://textual.textualize.io/guide/testing/) describes
20+
`run_test()` as headless app execution; it does not turn lifecycle behavior off.
21+
The [Pilot API](https://textual.textualize.io/api/pilot/) defines `pause()` as a
22+
way to wait for pending messages and CPU-idle settlement. It is not a generic
23+
delay to delete. The [Worker API](https://textual.textualize.io/api/worker/)
24+
documents worker state and cancellation as lifecycle behavior to test directly.
25+
26+
Use `AirlockApp(test_harness=True)` explicitly and only for tests that assert
27+
the production pane tree, static rendering, or deterministic navigation while
28+
not asserting lifecycle effects. Keep normal `AirlockApp()` mode for worker and
29+
timer lifecycle, cancellation, shutdown, stale callbacks, JSONL and MCP
30+
integration, and actual refresh behavior. Do not introduce a global harness
31+
fixture, remove `pause()` without a deterministic replacement, add xdist,
32+
retries, or relaxed timeouts merely to reduce duration.
33+
34+
## Evidence and coverage mapping
35+
36+
The baseline at `eb75a44` had 61 `run_test()` contexts: 10 explicit harness
37+
uses, 51 normal-mode mounts, and 48 `pilot.pause()` calls. The Slice 95 tree
38+
has 62 contexts, 18 explicit harness uses, 44 normal mounts, and the same 48
39+
pauses. The delta is a new normal-mode regression plus eight deliberate
40+
per-test harness choices.
41+
42+
The harness migrations are:
43+
44+
- `TestTUIBasic.test_overview_is_default`
45+
- `TestTUIBasic.test_overview_has_widgets`
46+
- `TestTUIBasic.test_guards_screen_exists`
47+
- `TestTUIBasic.test_logs_screen_exists`
48+
- `TestTUIBasic.test_config_screen_exists`
49+
- `TestTUIBasic.test_test_screen_exists`
50+
- `TestTUINavigation.test_navigation_by_number_keys`
51+
- `TestTUINavigation.test_all_five_views_accessible`
52+
53+
`TestTUIBasic.test_app_instantiates` remains an unchanged non-mount fixture.
54+
The new normal-mode
55+
`test_overview_stale_refresh_callback_is_ignored` captures the raw refresh
56+
worker callback, proves the pane remains mounted, makes only the providers
57+
table lookup raise `NoMatches`, and verifies the callback returns safely. Its
58+
RED proof on the parent of the guard change raised `NoMatches`; current GREEN
59+
coverage protects the existing teardown race. The named normal-mode contract
60+
`test_default_app_keeps_mount_lifecycle_enabled` remains in place, alongside
61+
existing lifecycle, JSONL/MCP, refresh, cancellation, and shutdown coverage.
62+
63+
## Measurement and outcome
64+
65+
The experiment used Python 3.12.3, Textual 6.2.1, pytest 9.1.1, Linux
66+
7.0.0-28-generic x86_64, the locked dependencies, and base `eb75a44`. Its only
67+
test delta is the stable Phase4 patch ID
68+
`558a8f37a11f8d7ca3698febcb387ac281fc9c4a`. One warm-up per side preceded
69+
three recorded A/B/B/A/A/B runs of:
70+
71+
```bash
72+
timeout --preserve-status 30s uv run python -m pytest \
73+
tests/harness/test_phase4_tui.py -q --durations=0 --durations-min=0
74+
```
75+
76+
All six retained runs passed nine tests without a timeout:
77+
78+
| Mode | Wall times (s) | Median (s) | Range (s) |
79+
| --- | --- | ---: | --- |
80+
| normal control | 14.33, 14.74, 13.65 | 14.33 | 13.65–14.74 |
81+
| explicit harness | 16.39, 14.65, 13.25 | 14.65 | 13.25–16.39 |
82+
83+
The explicit-harness median was 0.32s (about 2.2%) slower and had a wider
84+
range. The navigation tests and fixture setup dominated both modes. The result
85+
therefore validates only the safety and coverage mapping; it is **inconclusive
86+
for performance**. It authorizes neither a bulk harness migration nor a
87+
whole-suite speed claim.
88+
89+
CPU model, memory/load state, and raw historical CI duration logs were not
90+
retained. A new timing run can check whether this focused command still works,
91+
but cannot make past measurements reproducible. Any new candidate must have
92+
its own test-by-test coverage decision and controlled comparison.

‎dev/plans/0.5.15/95-tui-test-cycle-improvement-status.md‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
# Slice 95 — TUI-test-cycle improvement status
22

3-
**Status:** implementation, independent code review, and focused independent
4-
verification complete; ordinary-suite verification remains externally pending.
3+
**Status:** complete. Implementation, independent code review, and focused
4+
independent verification are complete; exact-head CI closes ordinary-suite
5+
verification.
56
The independent code review confirmed the normal-mode guard coverage and exact
67
eight-test mapping. Its timing-evidence ordering observation is addressed by
78
the explicit experiment-before-acceptance wording in the approved plan and the
@@ -108,11 +109,11 @@ support a speed claim or broader migration.
108109
Independent code review approved after three evidence-only FIX cycles. The
109110
independent verifier repeated focused GREEN, normal lifecycle, `make sync &&
110111
make verify`, Ruff, format, diff integrity, and strict MkDocs successfully.
111-
It found no implementation defect. Its exact ordinary-suite `make test` attempt
112-
was externally terminated at 14% with no pytest failure, timeout, or summary;
113-
the durable partial log is `/tmp/airlock-slice95-make-test.log`. A bounded
114-
`tests/test_tui.py` capture was terminated the same way. This is **inconclusive,
115-
not green**. A complete ordinary-suite result on the final commit (local durable
116-
runner or CI) is the sole remaining Slice 95 closeout condition. Rollback is a
117-
reviewed reversion of the two test files and this status record; there is no
118-
runtime or configuration rollback.
112+
It found no implementation defect. Its local ordinary-suite attempt was
113+
externally terminated at 14% with no pytest failure, timeout, or summary, so it
114+
was correctly recorded as inconclusive. Exact-head GitHub CI for commit
115+
`2a908c6` then passed every job: Docker (1m18s), docs (27s), lint (58s),
116+
security (17s), Gitleaks scan (7s), and the ordinary Python 3.12 suite (15m06s).
117+
That complete CI run closes the sole remaining condition. Rollback is a reviewed
118+
reversion of the two test files and this status record; there is no runtime or
119+
configuration rollback.

‎dev/plans/0.5.15/96-tui-test-performance-report-design.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,6 @@
11
# Slice 96 design — TUI test methods and outcome record
22

3-
**Status:** proposed for independent documentation/design review after Slice
4-
95 verification closes.
3+
**Status:** complete; independent documentation review and verification passed.
54

65
## Boundary
76

‎dev/plans/0.5.15/96-tui-test-performance-report-plan.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
# Slice 96 — TUI-test performance methods and outcome report
22

3-
**Status:** revised after Slice 96 audit; awaits Slice 95 independent
4-
verification before implementation. The report begins only after that closure,
5-
including the valid outcome that no material speedup was demonstrated.
3+
**Status:** complete after independent documentation review and verification.
4+
The valid outcome is that no material speedup was demonstrated.
65

76
## Purpose and inputs
87

‎dev/plans/0.5.15/96-tui-test-performance-report-status.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Slice 96 — TUI test methods and outcome report status
2+
3+
**Status:** complete. Independent documentation review and final verification
4+
passed.
5+
6+
## Re-evaluation and accepted scope
7+
8+
Slice 95 is closed: exact-head CI passed the ordinary Python 3.12 suite in
9+
15m06s and all companion jobs. Slice 96 therefore admits one developer-facing
10+
engineering note and an index link only. It does not change public/operator
11+
documentation, runtime behavior, dependencies, CI, or release notes.
12+
13+
## Delivered record
14+
15+
`dev/notes/tui-test-methods-0.5.15.md` records the authoritative baseline and
16+
current inventory, exact test mapping, normal-mode stale callback RED/GREEN,
17+
official Textual sources, controlled timing method, and result. It correctly
18+
states that the paired focused result is inconclusive for speed: normal median
19+
14.33s versus harness median 14.65s (0.32s / about 2.2% slower).
20+
21+
The note rejects bulk harness migration, pause removal, xdist, retries, and
22+
timeout relaxation. It preserves DFR-30/DAC-30 by separating pure
23+
composition/navigation tests from normal lifecycle and integration coverage.
24+
25+
## Verification and rollback
26+
27+
Independent review approved after one lifecycle-status correction. Independent
28+
verification passed `git diff --check`, the direct `dev/README.md` link,
29+
documentation contract (8 passed), strict MkDocs, the stale regression (1
30+
passed, 72 deselected), Phase4 (9 passed), and the retained normal-lifecycle
31+
harness tests (2 passed). The verifier also confirmed exact baseline/current
32+
inventory and timing arithmetic, direct Textual sources, and the absence of a
33+
performance promise. The known CI run `32037334617` is successful for exact
34+
commit `2a908c69cd12b1016ccf5c7150edeeca726b7c7f`, including `test (3.12)`;
35+
standard commit-list lookup did not rediscover it, but direct run inspection
36+
did. Rollback is a documentation-only correction or reversion.

‎dev/plans/0.5.15/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -224,8 +224,8 @@ authorize work before Slice 6.
224224
| 80 | CI review and improvement | Complete; preserve its least-privilege, pinned-action, and required-check controls in future changes. |
225225
| 90 | Release closeout | Complete and non-publishing; only delivered, verified slices are included and deferrals are explicit. |
226226
| 94 | TUI-test characterization and recommendation experiments | Complete; no bulk optimization was admitted from its non-causal timing evidence. |
227-
| 95 | Evidence-backed TUI-test-cycle improvements | In progress; bounded stale-callback coverage and paired focused timing are complete, with independent verification pending. |
228-
| 96 | TUI-test performance methods and outcome report | Draft; publish comparable before/after evidence and durable testing guidance. |
227+
| 95 | Evidence-backed TUI-test-cycle improvements | Complete; normal-mode stale-callback regression and eight pure-test harness mappings verified, with no performance claim. |
228+
| 96 | TUI-test performance methods and outcome report | Complete; independently reviewed developer methods/outcome note records comparable evidence and limitations. |
229229

230230
### Mandatory lifecycle for every included feature slice
231231

0 commit comments

Comments
 (0)