Skip to content

fast-usdc chain-impact test: implement the TODO assertions and trim its discarded CI cost #12747

Description

@turadg

What

packages/fast-usdc-deploy/test/chain-impact.test.ts is the dominant cost of the test-fast-usdc-deploy CI job (~18m on node-old, ~14m on node-new), and almost all of that cost produces data that CI discards while asserting nothing. There are two coupled problems:

  1. The test doesn't verify what it appears to. Its actual checks are unimplemented placeholders:

    292  test.todo('check that Exported object usage is flat');
    293  test.todo('check that Pending work (kernel promises) is flat');
    294  test.todo('check that vstorage usage is flat');
    295  test.todo('check that computron usage is flat');
    296  test.todo('check that heap usage is flat');
    

    The iterate simulation several times test collects observations (kernel stats, heap-snapshot sizes, vstorage/computron usage) but never asserts on them — the analysis is described in-comment as "we inspect $STATS_FILE somewhat manually."

  2. In CI it runs the full visualization-grade workload and throws the result away. It defaults to SUFFICIENT_ITERATIONS = REAP_PERIOD * 16 - 1 = 63 iterations with ~16 snapshotAllVats + reapAllVats + controller.dump() cycles, and CI sets neither STATS_FILE (so nothing is written) nor SIM_ITERS (so it runs the full count). The comment itself says 63 is for "plenty of data for visualization," while REAP_PERIOD * 2 + 1 = 9 "might be enough to observe stable results."

Net effect: CI pays ~11 minutes (the iterate simulation several times test alone) for what is effectively a smoke test — "does the chain-impact machinery run without throwing" — with no automated regression signal.

Why

  • CI cost / core utilization. In the baseline run 27784338369, test-fast-usdc-deploy (node-old) was ~18m and the second-slowest job in Test all Packages after test-portfolio-contract (since parallelized in test(portfolio-contract): parallelize the suite across CPU cores #12746). The single test iterate simulation several times has a cumulative marker of ~11m27s and is the last thing to finish — it runs alone on one core for ~10 minutes after the rest of the suite (including the heavy fast-usdc.test.ts) has completed at ~3.9m. AVA parallelizes across files, so one long serial test can't use the idle cores.
  • False sense of coverage. Because the assertions are test.todo, a real regression — unbounded heap/vstorage/computron/promise growth, which is exactly what this harness was built to detect — would not fail CI at 63 iterations or any other count. The expensive run buys no protection it appears to.
  • The expensive ops are pure measurement that CI discards. snapshotAllVats() (full XS heap snapshot of every vat, ~16×), controller.dump() at line 238, and getResourceUsageStats() (JSON.stringify of the entire vstorage map 3× per call, run every iteration) all feed the discarded observations array. reapAllVats GC and the sim.cleanup vstorage prune have functional value; the snapshot/dump/stringify do not, in CI.

Context: this surfaced while parallelizing packages/portfolio-contract tests (#12746) and surveying other suites for the same "one file/one test pins a single core" pattern. Unlike portfolio-contract (many independent tests in one file → split across files), this is one inherently-serial simulation loop, so the lever is the work it does, not file layout.

History: the value and the harness were introduced together in 9ad678b66a ("test: Fast USDC chain impact test", 2025-01-31) and have never been tuned since.

How

This needs judgment from the Fast USDC / deploy owners, not a unilateral trim. The assignee should decide and implement:

  1. Implement the test.todo assertions (lines 292-296). Define what "flat" means for each metric (exported objects, pending kernel promises, vstorage, computrons, heap), the threshold/tolerance, and the baseline source. This is the real point of the harness and the highest-value outcome.

  2. Decide CI's role for this test:

    • Smoke test (does the deploy + advance/mint + GC/prune machinery run): keep a lean version in CI.
    • Regression gate: requires the assertions above plus stats collection enabled in CI.
  3. Clean up CI cost consistent with (2). Candidate changes, to be confirmed by measurement:

    • Gate the measurement apparatus (snapshotAllVats, controller.dump() at line 238, getResourceUsageStats, observations.push) behind the existing writeStats/STATS_FILE signal, so CI skips it. The test already has if (writeStats) guards in two places (lines 127, 212, 282); extend that gating into the loop.
    • Set SIM_ITERS for the CI job (or lower the default) so CI runs a smoke-sized count rather than the visualization count.
    • Preserve full behavior when STATS_FILE/writeStats is set — this is the owners' manual analysis harness; the change must be "lean in CI, full when collecting stats," not "remove."
  4. Validate the per-op cost before committing. The loop already emits slog events (cleanup-begin/cleanup-finish, iteration-begin/iteration-finish via slogSender); a single timed run gives the exact split between snapshotAllVats, reapAllVats, and sim.iteration, confirming which lever (skip-snapshot vs. fewer-iterations) actually pays off.

Expected outcome: the iterate test drops from ~11m toward ~1-2m in CI, and CI gains a real assertion instead of a silent data-collection run.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Task.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions