Skip to content

Commit fa55403

Browse files
tonio-alucemaclaude
andcommitted
test(adapter-utils): make the next sandbox flake diagnosable
`execution-target-sandbox` has failed twice in CI and not once in several hundred local runs. This does not fix it. It makes the next occurrence carry its own evidence, because a third unreproducible failure would teach nothing. What is known. The observed signature was an empty stdout with exit code 0 - the child exited cleanly having produced nothing, which is what a lost stdin frame looks like from the test's side. Three mechanisms were checked and ruled out rather than assumed: - the helper resolving on `exit` rather than `close`, which would truncate output: a 200-iteration probe produced 0 truncations, and the failure was empty rather than partial anyway; - the wrapper reporting exit before stdout drains: it already listens on `close`, with a comment saying exactly why; - frame writes racing each other: the stream wrapper's `writeEvent` is synchronous and sequence-numbered. What is left needs to be observed where it happens. Two candidates remain and the runtime tree separates them: a stdin queue file still present means the host wrote the frame and the wrapper never consumed it; an empty queue with no output means it was consumed and the reply was lost on the way back. The report prints that tree, both proxy streams, the exit code, and the elapsed time - the last because the bridge and proxy run on 5s budgets that are generous locally and tight on a runner sharing a box with 19 other lanes, and a fast empty return is a different fault from one that nearly hit the ceiling. Attached to all four round-trip exchanges, not only the one that failed; they run the same protocol. Verified by forcing the assertion to fail and reading what it prints, rather than trusting that it would print something useful: a healthy run reports `elapsedMs=330` against that 5000ms budget and an empty stdin queue, which is the control the failing case will be read against. Deliberately not raising the timeouts. That would probably make the symptom go away, which is the reason not to do it blind - if there is a real ordering bug in the bridge, a longer budget hides it. adapter-utils typecheck clean; 44 pass, stable over five consecutive runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent add65ba commit fa55403

1 file changed

Lines changed: 86 additions & 11 deletions

File tree

packages/adapter-utils/src/execution-target-sandbox.test.ts

Lines changed: 86 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,22 @@ describe("sandbox adapter execution targets", () => {
148148
throw new Error(message);
149149
}
150150

151-
async function runProxyWithInput(command: string, input: string): Promise<{ stdout: string; stderr: string; code: number | null }> {
151+
type ProxyRunResult = {
152+
stdout: string;
153+
stderr: string;
154+
code: number | null;
155+
/**
156+
* How long the exchange took. The bridge and the proxy both run on 5s
157+
* budgets, which is generous locally and tight on a CI runner sharing a
158+
* box with 19 other lanes. A run that returns fast and empty is a
159+
* different fault from one that nearly hit the ceiling, and the numbers
160+
* are the only way to tell them apart after the fact.
161+
*/
162+
elapsedMs: number;
163+
};
164+
165+
async function runProxyWithInput(command: string, input: string): Promise<ProxyRunResult> {
166+
const startedAt = performance.now();
152167
const child = spawn(command, [], { stdio: ["pipe", "pipe", "pipe"] });
153168
let stdout = "";
154169
let stderr = "";
@@ -175,7 +190,61 @@ describe("sandbox adapter execution targets", () => {
175190
resolve(exitCode);
176191
});
177192
});
178-
return { stdout, stderr, code };
193+
return { stdout, stderr, code, elapsedMs: Math.round(performance.now() - startedAt) };
194+
}
195+
196+
/**
197+
* A failure report for a proxy exchange, attached to the assertions below.
198+
*
199+
* `execution-target-sandbox` has failed twice in CI and never once in a few
200+
* hundred local runs, so the next occurrence has to carry its own evidence -
201+
* a second unreproducible failure teaches nothing. The observed signature was
202+
* an empty stdout with exit code 0, meaning the child exited cleanly having
203+
* produced nothing, which is what a lost stdin frame looks like from here.
204+
*
205+
* The runtime tree is the part that discriminates. The stdin queue files are
206+
* written by the host and deleted by the wrapper once parsed, so what remains
207+
* says whether the frame was never written, written and never consumed, or
208+
* consumed normally and the reply lost on the way back.
209+
*/
210+
async function describeProxyRun(result: ProxyRunResult, runtimeRootDir: string): Promise<string> {
211+
const lines = [
212+
`proxy exit=${result.code} elapsedMs=${result.elapsedMs}`,
213+
`proxy stdout=${JSON.stringify(result.stdout)}`,
214+
`proxy stderr=${JSON.stringify(result.stderr)}`,
215+
];
216+
const walk = async (dir: string, depth: number): Promise<void> => {
217+
if (depth > 3) return;
218+
let entries;
219+
try {
220+
entries = await readdir(dir, { withFileTypes: true });
221+
} catch (error) {
222+
lines.push(`${" ".repeat(depth)}<unreadable ${dir}: ${(error as Error).message}>`);
223+
return;
224+
}
225+
for (const entry of entries.sort((a, b) => a.name.localeCompare(b.name))) {
226+
const full = path.join(dir, entry.name);
227+
if (entry.isDirectory()) {
228+
lines.push(`${" ".repeat(depth)}${entry.name}/`);
229+
await walk(full, depth + 1);
230+
continue;
231+
}
232+
// Small files are the queue and event frames, and their contents are
233+
// the point. Anything larger is a child script or a log; the size is
234+
// enough to say it exists.
235+
let detail = "";
236+
try {
237+
const raw = await readFile(full, "utf8");
238+
detail = raw.length <= 400 ? ` ${JSON.stringify(raw)}` : ` <${raw.length}B>`;
239+
} catch (error) {
240+
detail = ` <unreadable: ${(error as Error).message}>`;
241+
}
242+
lines.push(`${" ".repeat(depth)}${entry.name}${detail}`);
243+
}
244+
};
245+
lines.push(`runtime tree under ${runtimeRootDir}:`);
246+
await walk(runtimeRootDir, 1);
247+
return lines.join("\n");
179248
}
180249

181250
function combinedStream(
@@ -729,9 +798,10 @@ describe("sandbox adapter execution targets", () => {
729798

730799
try {
731800
const result = await runProxyWithInput(bridge!.agentCommand, "hello\n");
732-
expect(result.code).toBe(0);
733-
expect(result.stdout).toBe("out:hello\n");
734-
expect(result.stderr).toBe("err:hello\n");
801+
const report = await describeProxyRun(result, path.posix.join(rootDir, ".paperclip-runtime", "acpx"));
802+
expect(result.code, report).toBe(0);
803+
expect(result.stdout, report).toBe("out:hello\n");
804+
expect(result.stderr, report).toBe("err:hello\n");
735805
} finally {
736806
await bridge?.stop();
737807
}
@@ -1026,9 +1096,10 @@ describe("sandbox adapter execution targets", () => {
10261096

10271097
try {
10281098
const result = await runProxyWithInput(bridge!.agentCommand, "hello\n");
1029-
expect(result.code).toBe(0);
1030-
expect(result.stdout).toBe("out:hello\n");
1031-
expect(result.stderr).toBe("err:hello\n");
1099+
const report = await describeProxyRun(result, path.posix.join(rootDir, ".paperclip-runtime", "acpx"));
1100+
expect(result.code, report).toBe(0);
1101+
expect(result.stdout, report).toBe("out:hello\n");
1102+
expect(result.stderr, report).toBe("err:hello\n");
10321103
} finally {
10331104
await bridge?.stop();
10341105
}
@@ -1087,8 +1158,9 @@ describe("sandbox adapter execution targets", () => {
10871158
// frame flows, so it is observable as soon as the handle resolves.
10881159
expect(spanNames).toContain("sandbox.agentProcess");
10891160
const result = await runProxyWithInput(bridge!.agentCommand, "hello\n");
1090-
expect(result.code).toBe(0);
1091-
expect(result.stdout).toBe("out:hello\n");
1161+
const report = await describeProxyRun(result, path.posix.join(rootDir, ".paperclip-runtime", "acpx"));
1162+
expect(result.code, report).toBe(0);
1163+
expect(result.stdout, report).toBe("out:hello\n");
10921164
} finally {
10931165
await bridge?.stop();
10941166
}
@@ -1480,7 +1552,10 @@ describe("sandbox adapter execution targets", () => {
14801552
// Round-trip one input so a stdin-delivery control exec runs and gets
14811553
// recorded before the assertions below.
14821554
const result = await runProxyWithInput(bridge!.agentCommand, "hello\n");
1483-
expect(result.stdout).toBe("out:hello\n");
1555+
expect(
1556+
result.stdout,
1557+
await describeProxyRun(result, path.posix.join(rootDir, ".paperclip-runtime", "acpx")),
1558+
).toBe("out:hello\n");
14841559

14851560
// Exactly one exec runs on the persistent session: the long-lived agent
14861561
// command. It streams its output through the session log stream, so it

0 commit comments

Comments
 (0)