feat(agentry): add conflict-rebase git code-mode eval scenario

[0xpatrickbot]

Refs: #525

Description

Adds a conflict-rebase git code-mode eval scenario for @endo/agentry/eval. The scenario provisions a feature branch with one commit that conflicts on app.txt and one clean note commit, asks the agent to rebase onto integration, and scores only the final repository outcome.

The fixture precomputes the successful rebase by running the real conflict resolution once, capturing the expected feature tree and replayed commit patches, then resetting the repository so the eval starts on feature/conflict-rebase. The scorer checks that integration is still the same tip and is now an ancestor, the feature commits replay in their original order with fresh oids, the caller-supplied post-resolution patches match, app.txt has the requested combined wording, both notes are present, the worktree and index are clean, no rebase is left in progress, and HEAD is on the feature branch.

This replaces the earlier clean-rebase-only iteration with a deeper row: the clean replay case remains covered by the second feature commit, while the first commit exercises a real conflict resolution. That keeps the eval set smaller while making the rebase scenario more representative.

Security Considerations

The scenario exercises existing writable git authority: starting and continuing a rebase, writing a resolved file through the workspace, and staging it before continuing. It introduces no new capability surface or credential flow. The live eval remains gated by the existing ENDO_LLM_* / LAL_* environment contract.

Scaling Considerations

None. The scenario creates a small temporary git repository with a few commits and files, then tears it down with the test.

Documentation Considerations

The eval README now lists the conflict-rebase scenario and updates the no-credentials test description to cover multiple eval rows. There is no downstream data or deployment impact.

Testing Considerations

The no-credentials AVA tests cover the faux-provider pass path and negative scorer states for never rebased, wrong conflict resolution text, dropped feature commit, replayed commits out of order, and a dirty/conflicted rebase left in progress. Regression-test note: temporarily forcing the conflict-rebase scorer to return pass: true made those five negative tests fail with Value is not false: true, then the temporary change was reverted and the scenario tests passed again.

Compatibility Considerations

Additive. The existing stage-and-commit scenario and runner API continue to work; the shared scenario type is widened so eval rows can carry scenario-specific expected target shapes.

Upgrade Considerations

None.

[0xpatrickdev]

I didn't think this needed its own PR, but it's fine to leave the current split.

1. The second eval exposes some gaps in our file/folder structure. With everything in a single files (outcome.js,scenario.js, _eval fixture etc, files will grow to large and be harder to read and maintain. Wdyt about a folder per eval, each with outcome, scenario, etc, that seems better.

2. Let's have a designer spend some time on scenario design. My direction was "something with rebasing or worktrees". Let me be more concrete:

• these evals are specifically about the @endo/agentry harness: system prompt, tools, exos available in code mode
• the git evals are mainly to ensure we've implemented enough api functionality to be useful - it's be great if we can source gaps
• in the future we might have evals for testing that agents understand E, petNames, and other runtime specifics, but that's out of scope for this PR
• a few scenarios with deeper breadth are more valuable than a laundry list of the basics (rebase with no conflicts in the PR's current iteration).
• Note: we have tool gaps for richer scenarios (git remotes, npm registry, testing and linting code), but that's ok. The goal of this work is to expose so we can prioritize

[0xpatrickbot]

Must fix before un-draft

• [must-fix-loop] packages/agentry/src/eval/scenarios/rebase/outcome.js:72 resolves expected.ontoBranch after the agent has already run, so the target base is mutable by the model. A bad run can move or recreate main, then put feature on top of that new tip with the expected summaries and file contents; onto-is-ancestor will still pass even though the original onto tip was not preserved. Capture the pre-run onto tip OID in the fixture and make the scorer assert that exact commit is still the onto branch tip and the ancestor/base used for the replay. [rule: roles/jurors/assessor/AGENT.md]

• [must-fix-loop] packages/agentry/src/eval/scenarios/rebase/outcome.js:136 only checks a whitelist of expected paths at the final feature tip. A run can add an extra tracked file, or otherwise change unchecked tree state, while still matching the selected file contents and keeping the worktree clean. Since this is an outcome scorer, it needs an exact tree or diff invariant for the expected post-rebase state, not only positive assertions for a few paths. [rule: roles/jurors/prover/AGENT.md]

Should fix in this PR

• [summary-fix] packages/agentry/src/eval/scenarios/rebase/outcome.js:100 checks replay identity through summaries plus fresh OIDs, but it does not prove the replayed commits carry the original patches. A run could reset feature to main, create a fresh empty commit with the first expected summary, then commit the final feature.txt content with the second expected summary; the current summary, OID, final file, clean-tree, and branch checks can all pass without preserving the original per-commit patch sequence. Either compare each replayed commit's patch or tree against the original feature commit sequence, or narrow the documented scenario contract to final-tree-plus-subject matching. [rule: roles/jurors/wire-watcher/AGENT.md]

[0xpatrickbot]

Addressed the 2026-06-30 panel findings in 744c0ec:

• must-fix-loop mutable onto tip: the rebase fixture now captures the pre-run onto tip oid, and the scorer asserts the onto branch still points at that exact oid and slices the replay above that captured base. Added a regression where main is moved before replaying.
• must-fix-loop partial tree whitelist: the fixture now records the expected post-rebase feature tree oid, and the scorer asserts the final feature tree oid exactly matches it. Added a regression where the replay adds an extra tracked file.
• summary-fix patch identity: the scorer now compares the replayed commits' per-commit diffs against the original feature commit diffs in order. Added a regression where summaries and final tree match but the original patches were not replayed.

There were no inline review threads exposed for the 0xpatrickbot review, so I am replying on the PR with the per-finding update.

[0xpatrickbot]

Reworked this draft to address the 2026-06-25 review by replacing the clean-rebase-only row with the deeper conflict-rebase scenario. The clean replay case is still covered inside the scenario by the second feature commit.

[0xpatrickbot]

Garden panel verdict

Panel: code-panel. Round: 1.

Findings

• [must-fix-loop] eval-export-docs | Document the three new conflict-rebase eval exports and their roles. [rule: roles/jurors/surfacer/AGENT.md § Operating norms]
• [summary-fix] conflict-scenario-target | Copy only declared target fields into the scenario instead of hardening and exposing the provisioned repository object. [rule: roles/jurors/purist/AGENT.md § Side-channel closure and Family-consistency]
• [summary-fix] pipelined-file-lookup | Chain nested lookup calls with E(nodeP) so filesystem path traversal remains CapTP-pipelined. [rule: packages/platform/src/fs/extended/helpers.js]
• [summary-fix] integration-tip-negative-test | Add a test that moves integration after a correct rebase and verifies integration-branch-tip fails. [rule: skills/adversarial-tests/SKILL.md § Boundary]

[0xpatrickbot]

PR #526 reviewer context
Current head: 932c7eaab616822315072aa5b9790870667e504c

• Adds the @endo/agentry/eval conflict-rebase code-mode scenario, including repository provisioning, a declared target, and outcome scoring for the preserved integration tip and ancestry, replayed summaries and fresh commit IDs, per-commit patches, exact final tree and file contents, clean status, and completion on the feature branch.
• Adds scenario barrels, public JavaScript and TypeScript exports, README coverage, fixtures, focused no-credential tests and negative scorer cases, while retaining the stage-and-commit row; it also updates dependencies and the lockfile and regenerates the daemon-cas composite reference.
• Recorded verification for this head: agentry 83 tests passed, daemon 915 passed with 5 skipped, and git diff --check was clean.

[0xpatrickbot]

Live model eval results

The PR now contains the three split fixes:

• 132b4c05f fix(agentry): contain detached eventual-send failures
• 6e157e089 fix(exo-stream): contain reader pump rejections
• 7c0d60333 feat(agentry): persist live eval diagnostics

There are six distinct completed model campaigns in the retained evidence, so all six are listed below, one table per scenario, ordered by model.

conflict-rebase (3 of 6 passed)

Model	Result	Time	Tokens in / out / cache-read	Cost	Turns	Tool calls / errors
anthropic/claude-sonnet-5	✅ passed	1m00.4s	104,843 / 3,462 / 0	$0.244306	16	15 / 2
deepseek/deepseek-v4-flash	✅ passed	2m06.8s	62,007 / 3,643 / 88,576	$0.0073087	30	29 / 4
google/gemini-2.5-flash-lite	❌ failed outcome	13.3s	2,539 / 64 / 0	$0.0002795	1	0 / 0
google/gemini-3.5-flash	❌ failed outcome	28.5s	27,234 / 3,031 / 22,862	$0.0715593	13	12 / 6
nvidia/nemotron-3-ultra-550b-a55b:free	❌ failed outcome	54.4s	67,421 / 1,994 / 65,280	N/A	30	29 / 17
tencent/hy3:free	✅ passed ²	2m43.1s	12,303 / 7,573 / 132,864	N/A	20	19 / 1

stage-and-commit (6 of 6 passed)

Model	Result	Time	Tokens in / out / cache-read	Cost	Turns	Tool calls / errors
anthropic/claude-sonnet-5	✅ passed	8.2s	13,239 / 297 / 0	$0.029448	3	2 / 0
deepseek/deepseek-v4-flash	✅ passed	13.7s	9,467 / 633 / 2,432	$0.0009424	4	3 / 1
google/gemini-2.5-flash-lite	✅ passed	3.6s	6,041 / 377 / 4,304	$0.0007979	4	3 / 3
google/gemini-3.5-flash	✅ passed	3.6s	5,358 / 246 / 0	$0.010251	2	1 / 0
nvidia/nemotron-3-ultra-550b-a55b:free	✅ passed	29.4s	43,173 / 1,403 / 0	N/A	12	11 / 6
tencent/hy3:free	✅ passed	10.8s	3,344 / 360 / 4,864	N/A	3	2 / 0

Paid costs are calculated from the current OpenRouter catalog prices and recorded input, output, and cache-read usage. Free-model cost is shown as N/A.

² The Tencent conflict-rebase result is from a 2026-07-11 rerun. The run now passes under the standard budget, which this job raised to 5 minutes, completing all eleven outcome checks in 2m43.1s. The original 2-minute "timed out" row was a budget artifact, not a model failure: the event stream showed steady tool-call progress when killed, not hanging. An earlier Tencent run had also reached both passing outcomes but exited afterward because of reader-pump unhandled rejections (since fixed in this PR).

Failure analysis

• NVIDIA conflict-rebase: the model started a rebase but left it unresolved. It tried invalid capability shapes including git.status({}), git.filesystemAt({path: "app.txt"}), git.filesystemAt({branch: "feature/conflict-rebase"}), and git.show({object: "HEAD:app.txt"}). Earlier NVIDIA evidence also showed workspace.named("app.txt") and a git.show({oid, path}) mismatch. The final repository was still mid-rebase, missing the feature sentence and feature note.

• Gemini 2.5 Flash Lite: it first used top-level await, then tried E.NamedEnt(...), then passed {path: "README.md"} directly to git.add() instead of using the required remotable entry from git.status(). It made no tool call for conflict-rebase.

• Gemini 3.5 Flash: it reached the rebase, but called open() without its required argument, tried unavailable Far(...), and passed a plain {close, write} object to streamBase64() instead of a declared remotable writer.

The post-fix NVIDIA reruns and both Gemini runs exited for ordinary outcome assertions, not unhandled promise or reader-pump crashes. The durable event artifacts retain sanitized API errors rather than full generated source.

[0xpatrickdev]

scenario-specific types should go in a types.ts file in that folder. Include an additional refactor commit to address existing stage-and-commit specific types.

Also - scenario specific code/types/consts should not be part of any public api surface in this package. Only eval building/testing machinery. Please adjust code added in this PR and add refactor commits for the existing scenario in this package.