Graph as the core artifact: per-correlation sealed recordings, aligned scoring, parallel driving — delivery plan

[maverox]

Why

A replay produces two execution graphs, and today the system constructs both and consults neither: the verdict classifies over flat call streams (divergence/mod.rs:2295 — "the verdict below is unaffected"), the dashboard rebuilds a tree from ledger span-path strings because a name-keyed merge of the raw graph forests pairs nothing (web/src/lib/spine.ts header), every run re-derives per-correlation structure from the raw tape, a run pulls the entire session (361 MB) to drive 3 correlations, and the kernel drives sequentially in record order although correlations are independent test cases by construction.

#60 (docs/design/graph-aligned-scorer.md) defines the comparison half: classification as alignment of two activation forests. This issue is the construction, storage, and delivery half, plus the full plan to land both without regressing the running system. The claim:

The activation forest is the recording's core artifact. The lowest unit of storage, selection, replay, and scoring is a single correlation: compaction materializes one forest per correlation with its boundary events attached, a run pulls exactly the correlations it drives, the kernel drives them in parallel in any order, and the scorer aligns forest against forest.

Target design

1. Graph as core artifact. Compaction materializes per-correlation activation forests — graph nodes, boundary events attached by (recording_run_id, graph_node_id), a named unattached annex, per-node subtree-event-count rollups — with balanced accounting asserted: every event attached or in the annex, every node in exactly one correlation forest or the ambient forest.
2. Per-correlation storage. sessions/v1/{id}/correlations/{corr}/{meta.json, tape.ndjsonl.zst, forest.json}, an ambient/ sibling (uncorrelated traffic, always in scope), an index/ of concatenated meta heads for one-GET listing, and a manifest that accretes correlation entries with per-correlation content hashes, written last. Meta heads carry the request vocabulary already captured on root nodes (EXECUTION_GRAPH_SELECTED_FIELDS: flow, connector, payment method, merchant, tenant, status) plus counts, gseq range, completeness, and seedability — the picker filters on metadata instead of matching ids.
3. Additive, early compaction. A tailing compactor seals correlations as they complete (ingress finalizer + quiesce window), while the session is still recording. Late arrivals reseal that one correlation loudly (meta.completeness); correlations overlapping drop-gaps carry possible_gap: true.
4. Runtime substrate first. Graph nodes state their task lineage; the tape's two sequence spaces are named and drop accounting becomes kind-aware; the record-only deja::http_incoming ingress span asymmetry — a live bug demoting rank-2 lookups on every replay today — is fixed at the producer; replay-side nodes carry run identity.
5. Aligned scoring (docs: graph-aligned scoring — classification as alignment of two execution trees #60 implemented). detect() switches per correlation: graph-aligned when both forests exist, the span-scoped flat scorer otherwise, with scoring_mode declared per correlation. The call ledger regenerates from the alignment, retiring the pre-fix(scorer): pairing is span-scoped and runs after every resolution is known #57 pairing identity and the "may be a mis-pairing" UI caveat.
6. Parallel driving. Bounded concurrency across correlations, record order within one; KERNEL_PARALLELISM from the environment, default 1. The standing invariant is order-invariance: the same scope at K=1 recorded order and K=8 shuffled must produce identical verdicts and per-correlation outcomes.
7. Explicitly out of scope. Serving identity stays the lookup ladder; the landing tier stays flat; no cross-correlation alignment; no old-tape backfill (old tapes take the declared flat tier; re-record rather than migrate).

The tracks

Sizes are S/M/L. Stacks go up with gh stack. Every PR leaves the system green on its own; a PR whose behavior flips a default sits behind an environment gate with a declared fallback.

Track A — runtime substrate

• A1 (M) feat(graph): stamp task lineage onto execution-graph nodes at close — every tape-carried node states its own bucket_id/fork_seq, read at span close so layer registration order cannot race it; v8→v9, serde-additive. After G2 pins the v8 fixture.
• A2 (M) fix(writer): name the tape's sequence spaces and make drop accounting kind-aware — the boundary and graph counters stop sharing one drop ledger; markers gain per-space truth additively (ranges stays boundary-only and byte-compatible; graph_ranges is a new key), closing the doc/code contradiction in the compatible direction. After G2.
• A3 (M) fix(ingress): replay drives requests under the same deja::http_incoming span as record — kills the systematic rank-2→rank-3 demotion and gives the replay forest the same ingress root the record forest has. Vendor-fork commit. Gate includes a before/after resolved_by_rank diff on the same tape, and a check that the oldest tape any environment still replays was recorded with the prefix (pre-prefix tapes are re-record-only, stated in the PR body).
• A4 (S) fix(replay): stamp run identity onto replay-side graph nodes — symmetric attach identity for the forest builder; consumed defensively (see C5).
• A5 (S) vendor pin bump — the explicit deployment step that activates A1/A2/A4 in candidate images. Owned here, gated on demo/run-self-check.sh 9/9 and the A3 rank histogram check. Without this item the plan can be fully green in CI while real graph-tier runs misalign — nothing else in any track owns it.

Track B — shared forest builder + per-correlation compaction

• B1 (M) refactor(forest): extract activation-forest resolution primitives into a new deja-forest crate — parent map, root walking with dangling-parent promotion and the cycle guard, anchor closure, capture-vs-delivery gap classification; one shared home for compactor and orchestrator, without weakening the ScopedRecording/TapeSlot seam (the scope_invariant test survives).

• B2 (M) feat(forest): shared builder — (nodes, events) → per-correlation forests, annex, rollups, balanced accounting — the pure function both sides call; construction fails on imbalance. The golden fixture pins the join contract: on a single-run stream, attach keys on graph_node_id alone, with run identity as a loud assertion rather than a join column (so the replay side works both before and after A5).

• B3 (L) feat(compactor): per-correlation sealed layout (manifest v2) — correlation objects, meta heads, index, manifest last; v1 sessions stay readable as the declared flat tier. Ports G4's tier-equivalence test to the v2 path as part of the deletion of seal_session. Decides index shape (additive fields on CorrelationSummary rows) before F4's type mirror lands.

• B4 (L) feat(compactor): tailing incremental compaction — finalizer + quiesce, late-arrival reseal, possible_gap, conditional-PUT manifest accretion. Pins the demo MinIO image tag; the startup self-test distinguishes "store ignores preconditions" (refuse) from "client never sent one" (config bug).

• B5 (M) feat(s3): read the per-correlation sealed form — the v2 pull path plus the readers Track C needs. The in-process seal fallback goes behind DEJA_SEAL_LAYOUT (default v1) so B5 does not silently become the fleet's first v2 writer; an unknown manifest_version degrades to a declared landing-rescan fallback, never a hard parse error — otherwise a v2-sealed recording is bricked for every pre-B5 reader and rollback is one-way. The one-way-rollback property is stated in the PR body.

• B6 (M) ops(compactor): compaction job image, single-writer contract, IAM — the tailer finally runs somewhere with the write permission the replay job was deliberately denied in fix(replay): stop a replay writing the tape store it is judged against #32 (s3:PutObject on sessions/v1/*); demo overlay runs it locally.

  Runtime shape: an incremental job over the S3 landing tier — deliberately not a Kafka consumer. Vector stays the only Kafka consumer; the landing objects are the compactor's input queue and the S3 list is its poll. Three properties force this:

  1. Landing is the accounting boundary. Ingest accounting requires an input that can be enumerated and re-read so every line becomes an output or a named drop; S3 objects are re-readable forever, Kafka offsets age out with retention.
  2. Seals are content-addressed and must be reproducible. seal_id hashes the raw input lines — re-sealing after a compactor fix only works if the input still exists byte-for-byte.
  3. A second consumer group is a second truth. Vector-with-acks defines what was durably recorded; a Kafka-consuming compactor could seal lines that never landed (or miss ones that did), diverging from the landing tier every fallback path rescans — the producer/consumer split this codebase's rules exist to kill.

  The latency argument for Kafka is illusory: Vector batches to S3 on its own cadence, and the quiesce window for late-closing spans dominates end-to-end freshness anyway; the target is "replayable minutes after recorded," which polling meets. Deployment: a Kubernetes CronJob (1–2 min, concurrencyPolicy: Forbid as belt-and-braces — the conditional-PUT manifest write is the real single-writer guard) with its own service account; the compose demo runs the same binary as a looping service since compose has no cron; the existing CLI batch mode remains the degenerate one-pass case for ended sessions. Each pass is a pure function of S3 state (no offsets, no local state), so a crash before the manifest accretion leaves only idempotently-rewritable orphan objects, and promoting to a long-running single-replica tailer later is a deployment change, not a code change.

Track C — orchestrator consumption

• C1 (S) fix(lifecycle): an empty record graph writes the note the scorecard reads, not stderr — standalone, lands immediately.
• C2 (M) refactor(lifecycle): choose the run's correlations from the sealed index, tape scan as the declared fallback — one GET instead of a full tape scan; precondition for C3.
• C3 (L) feat(orchestrator): pull only the scoped correlation objects + ambient — deps corrected to B3 + B5 + C2. Scoped pulls materialize into scope-keyed slots (recordings/{id}/scoped/{scope_hash}/…), never the shared canonical events.jsonl — a scoped tape in the shared slot is silently mistaken for the whole session by older readers and turns the today-benign concurrent re-pull race into a corrupting one. TapeSlot owns the new path so the seam grep still holds; the coverage sidecar is an attestation, not the guard.
• C4 (M) feat(orchestrator): ScopedRecording::forests() — the record graph becomes a slice of the seal — extract_record_graph consumes the sealed slice; on-the-fly extraction stays as the declared flat tier. Owns migration 0007 (both record_forest and replay_forest artifact kinds) so C5 cannot collide on the migration number. Gate includes the demo smoke (lifecycle change), not just verify alone.
• C5 (M) feat(lifecycle): materialize the replay-side forest from observed.jsonl through the shared builder — same code path as the record side; publishes the aligner's second input. Join per the B2 contract (node-id-only, run identity asserted), so it is correct during the A5 pin-lag window instead of annexing every event fleet-wide.
• C6 — folded into F6, which owns the /graph response schema; C-track only produces and hydrates the artifacts.

Track D — the graph-aligned scorer (#60)

• D1 (L) feat(scorer): pure graph aligner over activation forests, with golden alignment tests — skeleton pruning via stored rollups, k-th same-named-child alignment, IdentitySkew reconciliation through source_event_global_sequence, suspected-rename shape flag, asserted balanced accounting. Goldens built from the shared synthetic generator (G0): wall correlation, clean correlation, request-only 400, the same-named loop where serving FIFO and alignment k disagree (asserts the skew is reported, not scored).
• D2 (S) feat(scorer): load both execution graphs into RunArtifacts — record graph from the run's scoped artifact, replay graph split out of observed.jsonl in the same pass that yields observed calls.
• D3 (L) feat(scorer): per-correlation graph-aligned scoring tier with declared scoring_mode — deps gain E4 (the order-invariance test must precede it) and a defensive rule: a one-sided ingress root demotes that correlation to flat with reason ingress_root_asymmetry, so D3 is correct even before A3/A5 reach the candidate image. All existing counters kept as projections; counter_disagreements() still balances. Gate: demo smoke with E2E_EXPECT_SCORING_MODE=graph.
• D4 (M) feat(ledger): regenerate the call ledger from the alignment — retires the pre-fix(scorer): pairing is span-scoped and runs after every resolution is known #57 (correlation, boundary, method) pairing and the UnifiedView caveat it necessitated.
• D5 (M) feat(scorer): port rules A/B and race evidence onto graph facts — verdict-affecting: gate includes the orchestrator smoke, not just verify alone.

Track E — kernel parallel driving

• E1 (S) feat(kernel): declared drive-order contract — recorded default, seeded shuffle for adversarial runs; boot-env docs tell the truth.
• E2 (M) test(kernel): concurrent-drive integration harness — K=1 recorded vs K=8 shuffled yield identical per-correlation diff sets, in bare cargo test. The fixture must include null-correlation side-effect calls interleaved across correlations: uncorrelated traffic shares one global occurrence bucket, and the kernel's own comment says record order matters for it. The test adjudicates that case explicitly — either serving is fixed (shared-bucket occurrences consumed in source_event_global_sequence order) or the honest contract is pinned: K>1 refuses or loudly warns on tapes with uncorrelated side-effect traffic. K=1 stays the default until the K=2 smoke passes on a demo tape that demonstrably carries ambient traffic.
• E3 (S) fix(runtime): FileObservedSink counts every dropped line and fails flush loudly — the let _ = write_all(...) swallow is a silent drop that would surface as fabricated omissions; fixed and pinned line-atomic under concurrent writers. Owns the sink test (absorbs the overlapping safety-net item).
• E4 (M) test(orchestrator): scorer order-invariance golden — the single order_invariance.rs; detect() invariant to cross-correlation interleavings, with the within-correlation-order negative control. Merges before any D PR and before E2/E5.
• E5 (M) feat(orchestrator): kernel parallelism and drive order are run parameters — resolved at creation, passed explicitly, stamped in run.params; re-POSTing a row re-runs identically.

Track F — API + dashboard

• F1 (S) ci(web): vitest harness + first spine characterization tests — stacked on G1's web job (single owner of the first web CI job).
• F2 (S) fix(dashboard): mirror HttpDiff.transport_error and the missing scorecard counters — live gap: the UI heuristically sniffs status 0 today.
• F3 (M) fix(dashboard): demoted and inconclusive ledger kinds render as what they are, never as Matched — live bug: markOf() returns null for demoted kinds and the detail pane falls through to "values agree".
• F4 (S) fix(dashboard): the correlations client parses CorrelationSummary rows, not strings — live bug since feat(replay): seal a recording once, and bound a run to what it can list #28; lands immediately, after B3's index-shape decision is confirmed in writing so the row type is written once.
• F5 (M) feat(replay): correlations endpoint serves meta heads; picker filters by flow/connector/status — deps: B3 + B5 + F4.
• F6 (L) feat(replay): /runs/{id}/graph serves per-correlation forests + ambient, flat tier declared per correlation — deps: C4 + C5. Owns the response schema.
• F7 (M) feat(replay): /graph carries alignment classifications and scoring_mode, read from the persisted artifact, never recomputed — deps: D3 (+D4).
• F8 (L) feat(dashboard): the spine consumes forests and alignment; the ledger-string tree becomes the declared flat tier.
• F9 (M) feat(dashboard): the report leads with subtree findings and names its scoring mode — the design doc's "two facts, not two thousand" claim, made real in the UI.
• F10 (S) feat(dashboard): show a run's kernel parallelism from run.params — deps: E5.

Track G — safety net, CI, gates

• G0 (M) test-support: shared synthetic tape/forest generator — new, land first among test infra: a dev-only crate generating tagged-NDJSON tapes and node/event sets with forks, same-named sibling loops, seeded interleavings, drop markers, and late-closing spans. Without it, five tracks each build private fixture builders that go stale independently when A1/A2 change the wire — the exact producer/consumer split this codebase's rules exist to kill. Run-0810-class evidence lives in sandbox S3; synthetic shapes are the only goldens hosted CI can run. Consumed by D1, E2, E4, G2, B2.
• G1 (S) ci: web typecheck/build job + just targets — first web gate; F1 stacks on it.
• G2 (M) test: golden v8 tape, scorecard-v1, and manifest fixtures pin the decode contract — must merge before A1/A2 or the fixture pins nothing; generated from G0 with a hand-audited field check, plus one small non-regeneratable demo-sourced line set.
• G3 (S) test: compile the web Scorecard/ledger mirror against the golden fixtures — one fixture, two enforced copies (a byte-identity check keeps the web copy honest).
• G4 (M) test: sealed fast path and landing rescan read back the same recording — written against compact_session (the surviving path), since B3 deletes seal_session; B3 ports it to v2-vs-landing equivalence as part of the deletion diff.
• G5 — dissolved: the sink half is E3's, the scorer-permutation half is E4's. One owner per invariant.
• G6 (M) ci(smoke): the demo compose run becomes the declared end-to-end gate with asserted facts — jq-checked assertions and named extension flags (E2E_EXPECT_RANK2, E2E_EXPECT_CONDITIONAL_PUT, E2E_EXPECT_SCORING_MODE) instead of human-read logs. Pulled forward to land before the B5/C3/D3 merge wave; smoke output pasted in PR bodies from then on. The smoke needs compose + a live Stripe key and cannot run in hosted CI — that is a named property of the gate, not a surprise.
• G7 (S) docs: compatibility matrix, per-track merge gates, rollout/rollback contract — checked into the repo. One correction already known: the scorecard lives both as the FS artifact and as replay_runs.scorecard JSONB read back through the run row, so scorecard additivity must be pinned for both.

Merge order

Every wave leaves main releasable; the two behavior flips (storage layout, kernel parallelism) sit behind env flags with declared fallbacks.

1. Wave 0 — freeze before moving: G1 → F1 → G0 → G2 → G3 → G4 → C1 → F2 → F3 → F4 → E1 → E3 → E4.
2. Wave 1 — runtime stack: A2 → A1 → A4 → A3 (vendor) → A5 (pin bump, gated).
3. Wave 2 — storage stack: B1 → B2 → B3 → B4 → B5 (flag-gated) → G6 → B6.
4. Wave 3 — consumption: C2 → C3 → C4 → C5 → E2 → E5.
5. Wave 4 — scoring: D1 → D2 → D3 → D4 → D5.
6. Wave 5 — presentation: F5 → F6 → F7 → F8 → F9 → F10 → G7.

Risk register

• The unowned deployment step was the biggest hole in the original breakdown: A1/A2/A4 activate only after the vendor pin bump, and D3/C5/B4 silently depended on it. A5 now owns it, and D3/C5 are written to be correct during the lag window (defensive ingress-root demotion; node-id-only join).
• Storage cutover is one-way if unguarded. A v2 manifest written by an ungated fallback bricks the recording for every older reader. DEJA_SEAL_LAYOUT default v1 + tolerant unknown-version handling makes the failure mode a slow declared tier instead.
• Scoped tapes must be structurally invisible to old readers — scope-keyed slots, not a sidecar contract that only new readers honor. Several sessions share DEJA_RECORDING_ROOT on the dev box; the mixed-version hazard is real there, not just in rolling deploys.
• Parallel driving's stated premise has a known exception: null-correlation traffic is order-sensitive at K>1. The order-invariance fixture encodes it; the default stays K=1 until the contract is pinned and the K=2 smoke passes on a tape with ambient traffic.
• The safety net must not be built twice. Single owners: E3 (sink), E4 (order invariance), G1+F1 (web CI), F6 (/graph schema), C4 (migration 0007), G0 (fixtures).

Open decisions

• Whether serving fixes the shared null-correlation occurrence bucket (consume in source_event_global_sequence order) or the K>1 contract excludes uncorrelated-traffic tapes — settled by E2's adjudication.
• Meta-head index shape: additive fields on CorrelationSummary rows (current lean) vs a replacement index — B3 decides, in writing, before F4 merges.
• Packing threshold for very large sessions (individual correlation objects vs packed parts with ranged GETs) — ship individual files, state the threshold in the design doc.
• The design doc itself (docs/design/unified-execution-graph.md, the construction/storage companion docs: graph-aligned scoring — classification as alignment of two execution trees #60's migration step 4 references) distills from this issue and lands with or before Wave 1.