You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
packages/fast-usdc-deploy/test/chain-impact.test.ts is the dominant cost of the test-fast-usdc-deploy CI job (~18m on node-old, ~14m on node-new), and almost all of that cost produces data that CI discards while asserting nothing. There are two coupled problems:
The test doesn't verify what it appears to. Its actual checks are unimplemented placeholders:
292 test.todo('check that Exported object usage is flat');
293 test.todo('check that Pending work (kernel promises) is flat');
294 test.todo('check that vstorage usage is flat');
295 test.todo('check that computron usage is flat');
296 test.todo('check that heap usage is flat');
The iterate simulation several times test collects observations (kernel stats, heap-snapshot sizes, vstorage/computron usage) but never asserts on them — the analysis is described in-comment as "we inspect $STATS_FILE somewhat manually."
In CI it runs the full visualization-grade workload and throws the result away. It defaults to SUFFICIENT_ITERATIONS = REAP_PERIOD * 16 - 1 = 63 iterations with ~16 snapshotAllVats + reapAllVats + controller.dump() cycles, and CI sets neither STATS_FILE (so nothing is written) nor SIM_ITERS (so it runs the full count). The comment itself says 63 is for "plenty of data for visualization," while REAP_PERIOD * 2 + 1 = 9 "might be enough to observe stable results."
Net effect: CI pays ~11 minutes (the iterate simulation several times test alone) for what is effectively a smoke test — "does the chain-impact machinery run without throwing" — with no automated regression signal.
Why
CI cost / core utilization. In the baseline run 27784338369, test-fast-usdc-deploy (node-old) was ~18m and the second-slowest job in Test all Packages after test-portfolio-contract (since parallelized in test(portfolio-contract): parallelize the suite across CPU cores #12746). The single test iterate simulation several times has a cumulative marker of ~11m27s and is the last thing to finish — it runs alone on one core for ~10 minutes after the rest of the suite (including the heavy fast-usdc.test.ts) has completed at ~3.9m. AVA parallelizes across files, so one long serial test can't use the idle cores.
False sense of coverage. Because the assertions are test.todo, a real regression — unbounded heap/vstorage/computron/promise growth, which is exactly what this harness was built to detect — would not fail CI at 63 iterations or any other count. The expensive run buys no protection it appears to.
The expensive ops are pure measurement that CI discards.snapshotAllVats() (full XS heap snapshot of every vat, ~16×), controller.dump() at line 238, and getResourceUsageStats() (JSON.stringify of the entire vstorage map 3× per call, run every iteration) all feed the discarded observations array. reapAllVats GC and the sim.cleanup vstorage prune have functional value; the snapshot/dump/stringify do not, in CI.
Context: this surfaced while parallelizing packages/portfolio-contract tests (#12746) and surveying other suites for the same "one file/one test pins a single core" pattern. Unlike portfolio-contract (many independent tests in one file → split across files), this is one inherently-serial simulation loop, so the lever is the work it does, not file layout.
History: the value and the harness were introduced together in 9ad678b66a ("test: Fast USDC chain impact test", 2025-01-31) and have never been tuned since.
How
This needs judgment from the Fast USDC / deploy owners, not a unilateral trim. The assignee should decide and implement:
Implement the test.todo assertions (lines 292-296). Define what "flat" means for each metric (exported objects, pending kernel promises, vstorage, computrons, heap), the threshold/tolerance, and the baseline source. This is the real point of the harness and the highest-value outcome.
Decide CI's role for this test:
Smoke test (does the deploy + advance/mint + GC/prune machinery run): keep a lean version in CI.
Regression gate: requires the assertions above plus stats collection enabled in CI.
Clean up CI cost consistent with (2). Candidate changes, to be confirmed by measurement:
Gate the measurement apparatus (snapshotAllVats, controller.dump() at line 238, getResourceUsageStats, observations.push) behind the existing writeStats/STATS_FILE signal, so CI skips it. The test already has if (writeStats) guards in two places (lines 127, 212, 282); extend that gating into the loop.
Set SIM_ITERS for the CI job (or lower the default) so CI runs a smoke-sized count rather than the visualization count.
Preserve full behavior when STATS_FILE/writeStats is set — this is the owners' manual analysis harness; the change must be "lean in CI, full when collecting stats," not "remove."
Validate the per-op cost before committing. The loop already emits slog events (cleanup-begin/cleanup-finish, iteration-begin/iteration-finish via slogSender); a single timed run gives the exact split between snapshotAllVats, reapAllVats, and sim.iteration, confirming which lever (skip-snapshot vs. fewer-iterations) actually pays off.
Expected outcome: the iterate test drops from ~11m toward ~1-2m in CI, and CI gains a real assertion instead of a silent data-collection run.
What
packages/fast-usdc-deploy/test/chain-impact.test.tsis the dominant cost of thetest-fast-usdc-deployCI job (~18m on node-old, ~14m on node-new), and almost all of that cost produces data that CI discards while asserting nothing. There are two coupled problems:The test doesn't verify what it appears to. Its actual checks are unimplemented placeholders:
The
iterate simulation several timestest collectsobservations(kernel stats, heap-snapshot sizes, vstorage/computron usage) but never asserts on them — the analysis is described in-comment as "we inspect$STATS_FILEsomewhat manually."In CI it runs the full visualization-grade workload and throws the result away. It defaults to
SUFFICIENT_ITERATIONS = REAP_PERIOD * 16 - 1 = 63iterations with ~16snapshotAllVats+reapAllVats+controller.dump()cycles, and CI sets neitherSTATS_FILE(so nothing is written) norSIM_ITERS(so it runs the full count). The comment itself says 63 is for "plenty of data for visualization," whileREAP_PERIOD * 2 + 1 = 9"might be enough to observe stable results."Net effect: CI pays ~11 minutes (the
iterate simulation several timestest alone) for what is effectively a smoke test — "does the chain-impact machinery run without throwing" — with no automated regression signal.Why
test-fast-usdc-deploy (node-old)was ~18m and the second-slowest job in Test all Packages aftertest-portfolio-contract(since parallelized in test(portfolio-contract): parallelize the suite across CPU cores #12746). The single testiterate simulation several timeshas a cumulative marker of ~11m27s and is the last thing to finish — it runs alone on one core for ~10 minutes after the rest of the suite (including the heavyfast-usdc.test.ts) has completed at ~3.9m. AVA parallelizes across files, so one long serial test can't use the idle cores.test.todo, a real regression — unbounded heap/vstorage/computron/promise growth, which is exactly what this harness was built to detect — would not fail CI at 63 iterations or any other count. The expensive run buys no protection it appears to.snapshotAllVats()(full XS heap snapshot of every vat, ~16×),controller.dump()at line 238, andgetResourceUsageStats()(JSON.stringifyof the entire vstorage map 3× per call, run every iteration) all feed the discardedobservationsarray.reapAllVatsGC and thesim.cleanupvstorage prune have functional value; the snapshot/dump/stringify do not, in CI.Context: this surfaced while parallelizing
packages/portfolio-contracttests (#12746) and surveying other suites for the same "one file/one test pins a single core" pattern. Unlike portfolio-contract (many independent tests in one file → split across files), this is one inherently-serial simulation loop, so the lever is the work it does, not file layout.History: the value and the harness were introduced together in
9ad678b66a("test: Fast USDC chain impact test", 2025-01-31) and have never been tuned since.How
This needs judgment from the Fast USDC / deploy owners, not a unilateral trim. The assignee should decide and implement:
Implement the
test.todoassertions (lines 292-296). Define what "flat" means for each metric (exported objects, pending kernel promises, vstorage, computrons, heap), the threshold/tolerance, and the baseline source. This is the real point of the harness and the highest-value outcome.Decide CI's role for this test:
Clean up CI cost consistent with (2). Candidate changes, to be confirmed by measurement:
snapshotAllVats,controller.dump()at line 238,getResourceUsageStats,observations.push) behind the existingwriteStats/STATS_FILEsignal, so CI skips it. The test already hasif (writeStats)guards in two places (lines 127, 212, 282); extend that gating into the loop.SIM_ITERSfor the CI job (or lower the default) so CI runs a smoke-sized count rather than the visualization count.STATS_FILE/writeStatsis set — this is the owners' manual analysis harness; the change must be "lean in CI, full when collecting stats," not "remove."Validate the per-op cost before committing. The loop already emits slog events (
cleanup-begin/cleanup-finish,iteration-begin/iteration-finishviaslogSender); a single timed run gives the exact split betweensnapshotAllVats,reapAllVats, andsim.iteration, confirming which lever (skip-snapshot vs. fewer-iterations) actually pays off.Expected outcome: the
iteratetest drops from ~11m toward ~1-2m in CI, and CI gains a real assertion instead of a silent data-collection run.