Skip to content

Commit 8a33f64

Browse files
adachi-440claude
andcommitted
ci: update deploy+backtest for the scenario contract, and cover the matrix path
The job still invoked `--regime calm-01 --blocks 12`. Renaming the regimes and making `--seed` required (ADR 0017 §1) broke it, and I missed it because the sweep I ran for stale references excluded `.github/`. The first re-run of the job failed earlier, in the deployer's Aave seeding, which this branch does not touch; that step passed on the retry, so it was a flake and the real failure was hiding behind it. While fixing the command, added coverage for the path the competition actually uses. `--scenarios` is a different code path from a single `--regime` run — it writes matrix.json / standings.json and has to keep going when one scenario fails — and nothing exercised it. Two scenarios is enough to run the loop, and the assertion checks that both scenarios produced results, that the standings ranked somebody, and that **both** metrics survive into matrix.json, since the scoring rule is expected to change and matrix.json is what makes a finished run re-scorable (#56). Verified locally against the state dump: both commands run and the assertion script passes on the real artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 8f5d03e commit 8a33f64

1 file changed

Lines changed: 32 additions & 1 deletion

File tree

.github/workflows/deploy-backtest.yml

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -87,8 +87,39 @@ jobs:
8787
npm run gen:local-constants
8888
npm run gen:state-dump
8989
90+
# A scenario is (regime, seed) and the regime YAML carries no seed, so --seed is required
91+
# (ADR 0017 §1). --blocks shortens the run for CI; the regime's own 360 would take 12 minutes.
9092
- name: Backtest against the dump
91-
run: npm run backtest -- --regime calm-01 --blocks 12
93+
run: npm run backtest -- --regime calm --seed 101 --blocks 12 --seconds 90
94+
95+
# The matrix path is what the competition actually runs, and it is a different code path from
96+
# a single --regime run: it writes matrix.json / standings.json and has to survive a scenario
97+
# that fails without abandoning the rest. Two scenarios is enough to exercise the loop.
98+
- name: Replay a small scenario matrix
99+
run: |
100+
cat > "$RUNNER_TEMP/ci-scenarios.yaml" <<'YAML'
101+
regimes: [calm]
102+
seeds: [101, 202]
103+
YAML
104+
npm run backtest -- --scenarios "$RUNNER_TEMP/ci-scenarios.yaml" --blocks 12 --seconds 90
105+
node -e '
106+
const { readdirSync, readFileSync } = require("node:fs");
107+
const dir = readdirSync("runs").filter((d) => d.startsWith("matrix-")).sort().at(-1);
108+
if (!dir) throw new Error("no matrix directory was produced");
109+
const m = JSON.parse(readFileSync(`runs/${dir}/matrix.json`, "utf8"));
110+
const s = JSON.parse(readFileSync(`runs/${dir}/standings.json`, "utf8"));
111+
const failed = m.scenarios.filter((x) => !x.agents);
112+
if (failed.length) throw new Error(`scenarios produced no result: ${JSON.stringify(failed)}`);
113+
if (m.scenarios.length !== 2) throw new Error(`expected 2 scenarios, got ${m.scenarios.length}`);
114+
if (!s.agents?.length) throw new Error("standings ranked nobody");
115+
// Both metrics must survive into the matrix, since the scoring rule is expected to
116+
// change and matrix.json is what makes a finished run re-scorable (ADR 0017 §4).
117+
for (const sc of m.scenarios)
118+
for (const a of sc.agents)
119+
if (!("netPnlUsdc" in a) || !("alphaUsdc" in a))
120+
throw new Error(`${sc.regime}#${sc.seed} ${a.id} is missing a metric`);
121+
console.log(`matrix ok: ${m.scenarios.length} scenarios, ${s.agents.length} ranked`);
122+
'
92123
93124
# The backtest exits 0 even when the scorer quietly read nothing, which is the failure mode
94125
# this whole job exists to catch, so assert on the run's own output.

0 commit comments

Comments
 (0)