Skip to content

fix(test): keep the microcompact scratch when a case FAILS, not when it passes - #346

Open
codeslake wants to merge 9 commits into
cnighswonger:mainfrom
codeslake:fix/test-scratch-survives-failure
Open

fix(test): keep the microcompact scratch when a case FAILS, not when it passes#346
codeslake wants to merge 9 commits into
cnighswonger:mainfrom
codeslake:fix/test-scratch-survives-failure

Conversation

@codeslake

@codeslake codeslake commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Retitled. It read "keep the microcompact scratch dirs off /tmp when a case fails", which is the opposite of what this does. The branch name was the honest one.

What a failing case leaves behind

proxy-microcompact-stability.test.mjs mints temp dirs for fixtures. A body that throws never reaches its own cleanup, so every dir is registered and removed once the file is done.

That removal was an after() hook, and after() runs after a red case too — so the directory holding whatever the case was looking at when it died was deleted before anyone could open it. The branch is named for keeping it; it did the opposite.

after() cannot know the outcome

Measured, four runtimes, on a file that HAS a failure and on one that does not:

node 18 node 20 node 22 node 24
process.exitCode inside after() 1 1 undefined undefined
process.exitCode at exit 1 1 1 1
same, on an all-green file undefined everywhere

The all-green row is the control: at exit the value discriminates, and it does so on every runtime CI runs. Inside after() it does not — on 22 and 24 the hook simply runs too early to know.

So the decision moves to process.on("exit"), and the removal becomes synchronous, because an exit handler cannot await and a promise-based rm would be registered and never run.

Where it lives

Registration moves to test/scratch-registry.mjsone new file, one new export (scratchDir). That is also what makes the lifecycle testable: it cannot be asserted in-process, because the decision is taken after every hook this file could run. The new case drives a real child both ways, and the passing arm is the control — without it, "the directory is there" is also what a registry that never cleans anything looks like.

A kept directory is named on stderr on the way out. Preserving one silently helps nobody.

Two holes in the guard beside it

The file carries a source-scanning guard: every temp dir must go through mcTemp().

  • It had no positive control. Neutering its pattern left it green over a file it no longer matched. The sample that proves the pattern still matches is assembled from pieces, so it is not itself a mint for the pattern to find.
  • Its premise was a runtime array this change empties, since the mint moved out of the file. It counts call sites in the source instead.

Verification

37/37 on the touched file, and green on 18 / 20 / 22 / 24.

reverted dies
always clean, ignoring the outcome a failing run deleted the scratch that is the only record of what it was looking at
never clean a passing run left its scratch behind — the registry cleans nothing
guard pattern neutered the pattern no longer matches a raw mint

2 files changed, +137 / −28. New files: 1. New exports: 1. New env vars: 0. New on-disk path shapes: mcprobe-*, from the new case's child.

Corrections to what is already written here

  • The commit message says the registry matches file-tmpdir.mjs "next to it". That file is on test: give each launcher-spawning file its own TMPDIR #347, not on this branch. The two are the same shape and are not yet siblings.
  • An earlier revision of this description showed the const scratch = [] + after(...) block as "the change". That block is what the change deletes.

@codeslake
codeslake force-pushed the fix/test-scratch-survives-failure branch from a120155 to 2a99715 Compare August 20, 2026 07:50
@codeslake
codeslake force-pushed the fix/test-scratch-survives-failure branch from 2a99715 to 3ba84fc Compare August 20, 2026 08:15
Fourteen cases each made their own `mc-` dir. Thirteen removed it on the last
line of the test body, so an assertion that threw skipped the removal; the
fourteenth (case 11) never removed it at all, so that one leaked on the green
path too. None of the fourteen was inside a try/finally — the file's three
`finally` blocks belong to withEnv, silenceStderr and silenceStderrAsync, and
restore env and stderr rather than remove anything.

Reproduced two ways: a clean run of the parent leaves 1 dir behind (case 11), and
one injected assertion failure leaves 2. After this change both are 0.

Registering the dir at creation and collecting in after() puts the cleanup in one
place instead of fourteen, and covers the paths that were missing rather than the
thirteen that already worked. The eager `rm` calls stay: they free disk during a
long run, and after() is idempotent with force.

after() does not cover SIGINT, SIGKILL, or a throw at import time — measured. The
first matters: Ctrl-C on a long suite still leaks. The last is vacuous here, since
every mcTemp() call is inside a test body.

test/proxy-wrapper.test.mjs already carries this shape; this is that pattern, not
a new one.

Co-Authored-By: Claude <noreply@anthropic.com>
@codeslake
codeslake force-pushed the fix/test-scratch-survives-failure branch from 3ba84fc to a48b69a Compare August 20, 2026 08:21
@codeslake
codeslake marked this pull request as ready for review August 20, 2026 09:20
codeslake and others added 5 commits August 26, 2026 02:59
…e cleanup

The registrar makes `after()` the single owner of every scratch dir, so the
13 per-case `rm` calls are dead: measured, the file runs in 84 ms and the 14
dirs it mints hold 66 KB together, so "frees disk during a long run" does not
hold. Two cleanup disciplines in one file is what produced the leak.

Nothing enforced the registrar. A new case pasting back the raw
`mkdtemp(join(tmpdir(), ...))` line strands its dir on any throw and the suite
stays green, which is how the 14th case leaked here unnoticed. Add the
source-level guard the sibling CA and wrapper suites already carry, with the
premise assertion so it cannot pass on an empty population.

Measured, isolated scratch root, 14 cases seeded to throw:
  base   green 1 stranded dir, failing 14
  branch green 0,               failing 0

Co-Authored-By: Claude <noreply@anthropic.com>
The guard excluded the registrar by the text `const d = await mkdtemp`, which
is a variable name, not a discriminator: a case writing `const d = await
mkdtemp(join(tmpdir(), "mc-"))` was excluded as if it were the registrar.
Measured with exactly that line in a case body: 36/36 pass and one dir
stranded, the leak the guard exists to report.

Hoist the prefix to a const so the registrar carries no string literal. The
pattern already requires one, so there is nothing to except and the hole
closes with the filter. Same line seeded again now fails the guard.

Co-Authored-By: Claude <noreply@anthropic.com>
…backtick mints

Two measured defects in the source-level guard, both from copying only
half of what the sibling guards do:

- A comment naming the anti-pattern reds the suite. Seeding a
  "never do: mkdtemp(join(tmpdir(), ...))" line into a case body failed
  the guard AT THAT COMMENT'S LINE. Documenting the wrong way broke the
  check that enforces the right way. Both sibling guards already skip
  comment lines; this one had kept only the self-escaping half.

- A backtick-quoted prefix escaped it entirely: 37/37 green while
  stranding a directory, which is the exact failure the guard exists to
  prevent. That shape is not hypothetical, it is already in this test
  suite as a mkdtempSync with an interpolated tag.

The quote class now covers all three JS string delimiters, and the
comment filter matches the siblings. Residual ceiling stated in place:
the scan is one line at a time, and a const prefix is the very thing
that excludes the registrar, so that one shape cannot be closed here.

Mutation-checked, four cases: raw mint caught, renamed variable caught,
backtick mint caught, comment no longer flagged. File 36/36 with zero
directories left under a private TMPDIR.

Co-Authored-By: Claude <noreply@anthropic.com>
…onestly

The comment filter covered one of JavaScript's three comment syntaxes.
Measured, both other styles still turned the suite red on prose that
merely NAMES the anti-pattern:

  /* never do: mkdtemp(join(tmpdir(), "mc-")) */      -> flagged
   * never do: mkdtemp(join(tmpdir(), "mc-"))         -> flagged

Same defect class the filter exists to close, half-closed. No file under
test/ uses block or JSDoc style today, so nothing was failing; this
closes the class rather than the one instance that had been measured.

The stated ceiling was not wrong, it was incomplete: it named two
residual shapes and there are at least five. A qualified os.tmpdir(), an
extra path segment, and an interpolated tag containing a quoted string
all pass too. The general sentence is both shorter and true, so the
enumeration is gone.

Also note at the declaration that SCRATCH_PREFIX is load-bearing.
Inlining the literal makes the guard flag its own registrar, and nothing
at that line said so.

Guard re-verified after the change: raw, renamed-variable, backtick and
single-quote mints all still caught; both comment styles now pass.
File 36/36 with zero directories left under a private TMPDIR.

Co-Authored-By: Claude <noreply@anthropic.com>
…ints

The previous commit widened the guard's comment filter from "//" to also
skip "/*" and "*". That was wrong, and measured wrong in two shapes that
went from CAUGHT to silently green:

  /* setup
  */ const _h = await mkdtemp(join(tmpdir(), "mc-"));   -> not flagged

  const obj = {
    *g() { const d = mkdtempSync(join(tmpdir(), "mc-")); return d; },
  };                                                    -> not flagged

The asymmetry is the whole point and the previous commit missed it:
nothing can follow "//" on a line, so skipping it can never hide code. A
block comment closes mid-line, and "*" also opens a generator method, so
skipping those lines hides real code sitting on them.

That traded a LOUD failure mode for a SILENT one. Prose that named the
anti-pattern turned the suite red: obvious, one line to fix, and nothing
leaked. A skipped mint leaks a directory with the suite green, which is
the exact failure this guard exists to catch.

The class it closed has zero instances anyway. No file under test/ uses
block or JSDoc comment style. Both sibling guards use the "//" filter,
so this also restores the cross-file consistency the branch argues for.

The ceiling sentence was false while that filter was in, since those
lines carry the exact call form the regex matches and escaped via the
line filter rather than the spelling. It is true again, and the reason
for "//" only is now stated so the next reader does not re-widen it.

Kept from the previous commit: the load-bearing note on SCRATCH_PREFIX
and the general ceiling sentence, both verified.

Re-verified: raw, renamed-variable, backtick, single-quote, block-close
and generator-method mints all caught. File 36/36, zero directories left
under a private TMPDIR.

Co-Authored-By: Claude <noreply@anthropic.com>
@vsits-codex-review-agent vsits-codex-review-agent Bot added changes-requested Blocking review findings are outstanding reviewed-by-codex-agent Directive/spec reviewed by Codex — no blocking findings labels Aug 26, 2026

@vsits-codex-review-agent vsits-codex-review-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #346 microcompact scratch lifecycle

Codex review: cross-LLM review, round 1.

Date: 2026-08-26
Reviewed: test/proxy-microcompact-stability.test.mjs at 7c62f0e918623bb9d93f3bb38c9cd0025174b6ff; PR #347 diff for interaction check
Round: 1
Label applied: changes-requested

Fork PR artifact note: the PR head is codeslake/claude-code-cache-fix, so per repo policy I did not push a committed review artifact to the contributor branch or to the base repo. This formal PR review is the artifact for this round.

What Is Correct

  • [Measured] CI is green on the reviewed head: test (18), test (20), test (22), GitGuardian, and Snyk all reported success for 7c62f0e918623bb9d93f3bb38c9cd0025174b6ff.
  • [Measured] The unchanged focused test file passes on my local runtime: node --version -> v24.11.1; node --test test/proxy-microcompact-stability.test.mjs -> 36/36 passing. After that success run, find /tmp -maxdepth 1 -type d -name 'mc-*' returned no entries, so the success-path cleanup works for the exercised file.
  • [Read] The new scratch names are minted by mkdtemp(join(tmpdir(), SCRATCH_PREFIX)) with constant prefix mc- at test/proxy-microcompact-stability.test.mjs:20-23; the path does not embed the session id or other request metadata.
  • [Read] PR #347 is a separate launcher-test isolation change: it adds test/file-tmpdir.mjs, sets process.env.TMPDIR per importing test file, and imports it in launcher/proxy tests. PR #346 does not set TMPDIR, does not add cleanup hooks outside test/proxy-microcompact-stability.test.mjs, and PR #347 does not touch the microcompact test file. The interaction shape is indirect only: if both changes later touched the same test process, tmpdir() would follow PR #347's TMPDIR; today they do not overlap in files or hooks.

Blockers

  1. [Measured] The failure-path scratch is still deleted, so the PR does not provide the post-mortem artifact it is meant to preserve.

    Code read: test/proxy-microcompact-stability.test.mjs:26-30 registers a file-level after() hook that removes every registered scratch directory. node:test still runs that hook after a failed test.

    Measurement: in an exported copy of the PR tree, I injected a review-only probe:

    test("forced failure preserves registered scratch probe", async () => {
      const dir = await mcTemp();
      console.error(`FORCED_SCRATCH=${dir}`);
      await writeFile(join(dir, "marker.txt"), "probe");
      assert.fail("forced failure for review probe");
    });

    Running node --test test/proxy-microcompact-stability.test.mjs failed as intended and printed FORCED_SCRATCH=/tmp/mc-KSM045, but the post-run checks reported:

    STATUS=1
    FORCED=/tmp/mc-KSM045
    MARKER=missing
    DIR=missing
    

    That is the opposite of the requested lifecycle: success scratch is cleaned, but genuine failure scratch must survive.

  2. [Read] The scratch location is still the process temp directory, not a controlled post-mortem location.

    The helper still calls tmpdir() directly at test/proxy-microcompact-stability.test.mjs:22. With no TMPDIR override, that is /tmp on the reviewed Linux environment; the forced-failure probe printed /tmp/mc-KSM045. The random suffix avoids collisions and the path name itself does not expose secrets, but the directory is not moved off the shared temp root and the PR does not document a default or precedence for any governing environment variable.

What Needs Attention

  • [Read] The guard test at test/proxy-microcompact-stability.test.mjs:782-799 catches only one textual spelling of raw mkdtemp(join(tmpdir(), "...")). It is useful as a narrow tripwire, but it should not be treated as proving that all future scratch creation is registered.

Bloat / Non-Functional

  • Production LOC: 0. Test-only diff: +54/-28 in one file. New files: 0. New exports: 0. New env vars: 0. New on-disk paths: no repo/product paths; test scratch remains under os.tmpdir().
  • Bloat finding: None. The size is proportionate to a test cleanup lifecycle change.

Recommendations

  • Gate cleanup on the test file's final result, or use an explicit success-only cleanup path that still runs after assertions complete but skips removal when any test in the file failed.
  • If the intended destination is not os.tmpdir(), introduce the directory-selection helper in the test code and document the default/precedence. If TMPDIR is the intended control point, say that explicitly and ensure the preserving failure path still prints or otherwise exposes the final scratch path to the operator.
  • Add a regression test that forces a failing child test file or subprocess and asserts the scratch directory remains after non-zero exit. A direct in-process assertion cannot prove this lifecycle because the check must happen after after() has run.

Bottom Line

Request changes. The success-path cleanup is measured working, but the failure path deletes the registered scratch before the operator can inspect it, and the path still lands under the process temp root unless external TMPDIR state says otherwise.

— Codex, cross-LLM review, round 1

…wn cleanup

The branch is named for keeping scratch across a failure and it did the
opposite: `after()` runs after a red case too, so the directory that holds what
the case was looking at when it died was removed before anyone could open it.

`after()` cannot know the answer. Measured on node 24: `process.exitCode` is
`undefined` inside a file-level `after()` on a file that HAS a failure, and `1`
by exit on that same file -- and `undefined` at both points on a file without
one, which is the control that makes the first reading mean something. So the
decision moves to `process.on("exit")`, where the outcome is settled, and the
removal is sync because an exit handler cannot await.

Registration moves to `test/scratch-registry.mjs`, matching `file-tmpdir.mjs`
next to it. That is also what makes the lifecycle testable: it cannot be
asserted in-process, because the decision is taken after every hook this file
could run, so the new case drives a real child both ways. The passing arm is
the control -- without it, "the directory is there" is also what a registry
that never cleans anything looks like.

Two things the guard beside it was missing, both now closed:
- a POSITIVE CONTROL. Neutering the pattern left it green over a file it no
  longer matched. The sample is assembled from pieces so it is not itself a
  mint for the pattern to find.
- its premise was a runtime array that this change empties. It counts call
  sites in the source instead, which cannot silently go to zero.

GREEN: 37/37
mutation: always-clean kills the failing arm; never-clean kills the passing arm;
  a neutered pattern kills the positive control; restored 37/37

Co-Authored-By: Claude <noreply@anthropic.com>
@codeslake codeslake changed the title fix(test): keep the microcompact scratch dirs off /tmp when a case fails fix(test): keep the microcompact scratch when a case FAILS, not when it passes Aug 26, 2026
@codeslake

Copy link
Copy Markdown
Contributor Author

Thanks — you are right about the head you reviewed, and it is fixed at a later one.

Reviewed: 7c62f0e9. Current head: 2a40964.

What changed. The file-level after() hook is gone. Registration moved to
test/scratch-registry.mjs, which removes on process.on("exit") and gates the
removal on the FILE's outcome:

process.on("exit", () => {
  for (const d of registered) {
    if (process.exitCode) { process.stderr.write(`[scratch kept] ${d}\n`); continue; }
    try { rmSync(d, { recursive: true, force: true }); } catch {}
  }
});

exit and not after() for the reason your review implies: after() runs too
early to know. Measured on node 24, process.exitCode is undefined inside a
file-level after() on a file that HAS a failure, and 1 by exit on the same
file. Removal is sync because an exit handler cannot await.

Measured, both directions, with your own forced-failure probe shape:

file outcome [scratch kept] line dir after exit
one test throws yes, with the path present
all pass no removed

The kept dir is named on stderr on the way out, since the runner's output is the
only place the caller will look for the path.

Could you re-review at 2a40964?

🤖 Generated with Claude Code

@codeslake codeslake closed this Sep 6, 2026
@codeslake codeslake reopened this Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changes-requested Blocking review findings are outstanding reviewed-by-codex-agent Directive/spec reviewed by Codex — no blocking findings

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant