[Architecture]: Build one end-to-end agent tool-journey conformance suite across every supported host

[morluto]

Priority

P1/P2 evaluation and release-engineering issue: Jacobian has strong component tests and many valuable model canaries, but no single suite verifies the complete agent journey from tool exposure through final evidence use across every supported host surface.

Area

MCP conformance, packaged connectors/skills, model-in-the-loop evaluation, regression governance, cross-surface telemetry.

Summary

Current regressions are discovered in separate layers and separate experiments:

server starts
  -> tools are exposed under the right identity
  -> agent discovers an applicable capability
  -> exact contract is understandable
  -> first execution call is valid
  -> error recovery succeeds when needed
  -> primary output is readable and correctly labelled
  -> complete resource is recoverable
  -> verifier receives the exact candidate
  -> assurance/scope are interpreted correctly
  -> final answer preserves the verified/computed values

The repository tests many individual arrows, but not the whole path under one frozen contract.

Examples of failures that passed neighboring layers:

• Fix MCP tool discovery over streamable HTTP #935 / Normalize root-union MCP output schemas on the SDK 2.0 boundary #1017: the server started, but an HTTP client could not complete tools/list.
• [Bug]: Keep canonical MCP tool identity intact through packaged connector aliases #1029: in-process canonical tools were correct, while the installed connector exposed incompatible names/schemas and impossible recovery guidance.
• Expose P-recursive evaluation under sequence domains #934 / Expose exact polynomial-power normalization workflow #936: applicable capabilities existed but were not found under natural agent vocabulary.
• Clarify inline verifier invocation envelopes #929 / Expose schema-valid invocation examples for exact replay verifiers #941: producer/checker contracts existed, but the agent constructed the wrong nested verifier envelope.
• Expose proof-critical projective flat previews #992 / [Superseded]: Do not add a primary-projection framework; return complete results or a small domain-specific summary #1033: a correct verified artifact existed, but proof-critical values were not usable in the immediate model-facing result.
• [Evaluation]: numeric-looking graph labels can misbind distance-matrix rows #930 / PR Bind graph distance-matrix entries to vertex labels #986: a verified matrix was semantically relabelled in the final answer.
• [Evaluation]: Successful exact matrix workflow reproducibly times out under selective grounding #938: exact backend work completed quickly, but the agent timed out after successful calls.
• [Bug]: Publish descriptor-owned assurance ceilings instead of inferring COMPUTED from EXPLORE #1067: runtime assurance was correct while the discovery card advertised a stronger ceiling.

No single component is sufficient to score these trajectories. “Tool initialized,” “tool was called,” “operation completed,” and “final answer was correct” are different gates.

Audit baseline: current main, including:

• tests/boundary/mcp/*
• tests/e2e/*
• tests/unit/tooling/test_codex_visibility.py
• benchmarks/tooling/codex_visibility.py
• benchmarks/tooling/codex_tool_context.py
• src/jacobian/eval/telemetry.py
• npm/cli-e2e.test.mjs

Current fragmentation

In-process conformance does not test installed surfaces

Boundary tests often use:

async with Client(create_server(...)) as client:

This is excellent for the canonical server object, but it cannot detect packaging aliases, stale managed Skills, launcher configuration, HTTP proxy behavior, or host-specific model-visible result projection.

Doctor certifies a narrow path

Doctor currently checks launch configuration, an older-protocol handshake, and tool names. It does not execute the full browse → inspect → invoke → invalid-retry → resource-read journey or compare schemas/results across hosts.

Visibility suites focus on adoption

The committed Codex visibility suites record discovery/invocation/verification/abstention signals. They are not a complete mathematical-output handoff suite and cannot prove that labels, ordering, candidate binding, resource consumption, or final claims are correct.

PR canaries are difficult to compare

Recent PRs use high-quality frozen theorem cases, but each defines its own prompt, model, output schema, telemetry summary, and success interpretation. One run per arm is often intentionally presented as a reproduction rather than a performance claim. The evidence is useful but does not form a stable release gate across contract changes.

Root cause

The test architecture mirrors implementation ownership rather than the agent’s actual workflow.

transport tests        own transport
schema tests           own schema
catalog tests          own descriptors
capability tests       own mathematics
evaluation tasks       own final answers
packaging tests        own installed files

An agent experiences all of them as one protocol. Without a cross-layer journey contract, a change can be locally correct while breaking the next handoff.

Proposed architecture

Build two complementary suites sharing one versioned journey schema.

1. Deterministic cross-surface protocol journey

No language model is required. Run canonical fixtures through:

1. in-process MCP;
2. stdio;
3. Streamable HTTP for every supported protocol version;
4. packaged Codex binding;
5. configured Claude/Cursor/Gemini/OpenCode launchers where automation is possible;
6. installed connector/host binding.

Each surface runs the same sequence:

list tools
browse
search
exact SUMMARY
exact CONTRACT
invoke returned example
submit one precise invalid request
apply returned recovery
read primary resource
invoke linked verifier when installed
inspect final result/record binding

Canonicalize and compare:

• tool canonical identity/local symbol;
• descriptions, annotations, execution metadata;
• input/output/contract digests;
• result discriminators and channel shapes;
• error code/path/rule/recovery;
• primary-result projection and resource links;
• scope, completeness, assurance, obligations;
• protocol version and tenant/policy/provider identities.

This suite catches implementation and packaging drift without depending on model sampling.

2. Repeated model-in-the-loop journey evaluation

Use frozen, transformed, and held-out tasks that require materially different paths:

• direct stable scalar operation;
• intent-led discovery with vocabulary mismatch;
• nested/discriminated payload construction;
• producer → inline verifier handoff;
• producer → artifact/resource verifier handoff;
• labelled matrix or ordered symbolic output;
• long-running/progress/cancellation case;
• weak/no-match fallback;
• negative control where Jacobian should be skipped.

Use repeated trials and report uncertainty. Hold model, reasoning effort, prompt bytes, task/verifier digests, policy, provider catalog, Skill condition, host, and protocol version fixed within comparisons.

Versioned journey event model

Record typed stages rather than reconstructing them from ad hoc transcript strings:

EXPOSED
DISCOVERY_REQUESTED
APPLICABLE_MATCH_RETURNED
CONTRACT_INSPECTED
EXECUTION_ATTEMPTED
EXECUTION_VALID_FIRST_CALL
RECOVERY_APPLIED
EXECUTION_COMPLETED
PRIMARY_RESULT_CONSUMED
RESOURCE_READ
VERIFIER_ATTEMPTED
VERIFIED_RECORD_BOUND
FINAL_VALUE_PRESERVED
FINAL_SCOPE_ASSURANCE_CALIBRATED

Events are observational only. They do not prescribe strategy and cannot authorize verification.

Required metrics

Report separately:

• availability versus available-but-unused;
• applicable discovery recall/precision;
• first-call validity and error-repair success;
• repeated searches and irrelevant calls;
• operation runtime versus time-to-next-action;
• text/structured/wire/model-visible bytes;
• resource-link follow-through;
• manual reconstruction/transcription;
• final and intermediate mathematical correctness;
• false certification and assurance/scope promotion;
• cancellation/timeout/late-worker behavior;
• prompt, cached/uncached, completion tokens and wall time.

Do not reward tool invocation itself.

Change-impact release gates

Map production changes to required journey lanes. Examples:

• MCP server/SDK/transport change → all deterministic surfaces + latest protocol lane;
• descriptor/schema change → contract/invocation/error lanes;
• output/result change → primary/resource/final-preservation lanes;
• Skill/packaging change → installed Codex + control without Skill;
• retrieval change → held-out discovery + abstention corpus;
• checker/assurance change → false-certification and exact-record lanes.

Avoid rerunning every expensive model suite on unrelated leaf mathematics. Use deterministic gates broadly and small preregistered model suites where behavioral evidence is necessary.

Evidence retention

Every report should bind:

• source revision/tree;
• package/Skill/connector artifacts;
• protocol and SDK versions;
• catalog/policy/provider/contract digests;
• task/prompt/output-schema/verifier digests;
• model/effort/host configuration;
• normalized events and raw trace/archive digest;
• exclusions, failures, and environment limitations.

Raw private traces may remain local, but checked-in summaries must identify exactly what was measured and avoid unsupported causal claims.

Acceptance criteria

• One deterministic fixture runs the complete list → browse → search → inspect → invoke → invalid-recover → resource → verifier path across every supported surface.
• Cross-surface comparisons use canonical tool and contract identities rather than local function-symbol equality alone.
• A model suite covers direct use, discovery, schema construction, result consumption, verification, long execution, no-match recovery, and abstention.
• Metrics separate server availability, adoption, valid execution, mathematical contribution, final correctness, and assurance calibration.
• Intermediate certificate/label/order correctness is scored, not only headline scalars.
• Repeated trials report uncertainty; one-off reproductions remain labelled as reproductions.
• Tool invocation is never itself a reward signal.
• Change-impact planning selects the smallest sufficient deterministic/model lanes.
• [Bug]: Keep canonical MCP tool identity intact through packaged connector aliases #1029, [Superseded]: Improve concrete examples/contracts; do not add prepared-call tokens or dynamic selected tools #1031, [Superseded]: Do not add a primary-projection framework; return complete results or a small domain-specific summary #1033, [Design]: Split mutating controls from pure math.run so MCP annotations are truthful #1034, [Architecture]: Replace ad hoc lexical tag patching with a versioned hybrid capability retrieval index #1059, [Bug]: Publish descriptor-owned assurance ceilings instead of inferring COMPUTED from EXPLORE #1067, [Bug]: Preserve field-level, machine-actionable diagnostics for math.find and tool-envelope validation errors #1071, [Architecture][Deletion]: Bound inline results; use one temporary MCP resource for oversized output #1074, [Compatibility]: Migrate remote MCP and doctor to the final 2026-07-28 stateless protocol lifecycle #1076, and [Architecture]: Make execution shape, deadlines, progress, and cancellation descriptor-owned #1077 each contribute at least one frozen acceptance fixture.
• Reports are digest-bound and distinguish host/environment limitations from product failures.

Related work

• [Evaluation]: Jacobian is available but unused across a four-case conjecture canary #845 owns available-but-unused evaluation evidence and recommends repeated held-out/negative-control work.
• [Evaluation]: Successful exact matrix workflow reproducibly times out under selective grounding #938 owns a successful-operation timeout reproduction.
• [Bug]: Keep canonical MCP tool identity intact through packaged connector aliases #1029 owns cross-surface canonical identity.
• [Compatibility]: Migrate remote MCP and doctor to the final 2026-07-28 stateless protocol lifecycle #1076 owns final-protocol conformance.
• Add a passive external reasoning observer #985 provides a passive external observer without changing the production contract.
• fix(benchmarks): harden Jacobian evaluation evidence #946 hardened evaluation evidence boundaries.
• [Feature]: Make capability contracts easier for agents to discover and compose #28 is the historical broad capability-ergonomics umbrella.

Non-goals

• Prescribing one mathematical research strategy or tool order for ordinary use.
• Treating model self-reports as proof of decisions.
• Replacing domain mathematical verifiers with an LLM judge.
• Requiring every leaf capability PR to run an expensive universal model benchmark.
• Publishing private chain-of-thought.
• Concluding that a treatment is causal from one trajectory per arm.