[Architecture]: Validate published invocation examples through the same full request preflight as math.run

[morluto]

Priority

P1/P2 agent-contract correctness issue: Jacobian describes invocation examples as validated and runnable, but installation-level validation is centered on the published JSON Schema. Domain/Pydantic cross-field invariants and canonical request preflight may reject an example later at math.run.

Area

Capability descriptors, invocation examples, JSON Schema, Pydantic/domain validation, installation, math.find CONTRACT views, verifier handoff.

Summary

Invocation examples are one of the highest-value agent ergonomics features in Jacobian. They let an agent move from an exact capability description to a correctly wrapped math.run call without reverse-engineering a nested schema.

The current common contract guarantees that an example:

• has a canonical JSON value;
• uses an advertised role (producer or checker);
• is validated against the descriptor's published input schema during installation.

That is not necessarily equivalent to the request boundary used at execution.

Repository documentation explicitly distinguishes:

JSON Schema remains the discovery contract;
Pydantic enforces cross-field conditions.

Examples of constraints that may live outside or beyond the portable schema projection include:

• dimensions/variable sets agreeing across fields;
• canonical ordering and normalization;
• digests binding exact content;
• domain closure or label coverage;
• mutually dependent optional fields;
• task/provider-specific preflight;
• inline verifier candidate/input consistency;
• operation-specific budget relations.

An example can therefore be structurally valid yet fail before mathematical execution. From the agent's perspective, that breaks the strongest advertised recovery path.

Audit baseline: current main as inspected on 2026-08-10.

Concrete evidence

Current descriptor validation is schema-oriented

CapabilityInvocationExample stores:

name
description
mode
input: dict[str, Any]

and requires canonicalizable JSON.

CapabilityDescriptor verifies uniqueness and that the example matches the descriptor role. Installation/catalog tests validate examples against the descriptor JSON Schema.

The descriptor does not carry one universal callable that proves the example passes the exact same domain/Pydantic request preflight used by its adapter at runtime.

JSON Schema is intentionally not the whole validation contract

docs/reference/tools.md states that domain adapters validate their complete Pydantic request model before computation or artifact writes, while JSON Schema is the discovery contract. It cites cross-field conditions such as polynomial-map dimensions and finite-table closure.

That is the correct trust boundary. It implies that schema-only example validation is not validation-equivalent for every capability.

Verifier examples are especially sensitive

Issue #941 and PR #929 show that weak models need concrete verifier examples for sibling input/candidate envelopes. A verifier example can be structurally well shaped but still fail because:

• candidate fields do not bind the input;
• dimensions/labels disagree;
• required result fields are internally inconsistent;
• the candidate is not canonical;
• a stored URI points to incompatible lineage;
• role-specific semantic preflight rejects it.

A published example that fails those checks would create another retry loop precisely where the example is intended to eliminate one.

Same-tree string tests are not sufficient

The managed skill and visibility tests validate selected literal payloads against current request models. That is useful but ad hoc. It does not establish a catalog invariant for every installed descriptor/example, plugin, provider feature set, or generated package.

Root cause

The system lacks one side-effect-free, validation-equivalent request preflight interface shared by:

runtime math.run dispatch
invocation-example installation
selected tool materialization
batch preflight
client/connector conformance

JSON Schema is being asked to serve both portable discovery and complete executable-contract validation, despite the architecture correctly relying on richer domain models for the latter.

Proposed architecture

1. Add a pure capability request preflight contract

Every installed adapter should expose or be wrapped by a side-effect-free operation such as:

preflight_request(
    *,
    # (role is determined by the capability ID, not a mode parameter)
    input: dict[str, Any],
) -> CanonicalValidatedRequest | tuple[CapabilityDiagnostic, ...]

Required properties:

• invokes the same complete request model/canonicalization used before runtime execution;
• performs capability/mode/input validation only;
• writes no artifacts, experiments, logs with payload values, or provider state;
• does not call the mathematical backend;
• returns the canonical validated request or bounded typed diagnostics;
• is deterministic for a fixed contract/version/configuration.

Runtime dispatch should call this same boundary rather than maintaining another validation path.

2. Validate every runnable example through preflight at installation

For each descriptor example:

1. validate portable JSON Schema;
2. invoke the exact side-effect-free preflight;
3. require canonical output stability;
4. ensure the mode and capability version agree;
5. bind the example to the input-schema/preflight contract identity;
6. fail capability registration when a purported runnable example is rejected.

Do not silently drop a bad example while leaving the capability installed unless the descriptor explicitly classifies the example as optional and the catalog reports the omission diagnostic. Prefer fail-fast for built-in reviewed examples.

3. Distinguish runnable examples from illustrative templates

Some operations require a URI or producer result unavailable in a static installation fixture. Do not publish placeholders as though they are executable.

Use finite kinds, for example:

RUNNABLE
TEMPLATE_REQUIRES_VALUE
TEMPLATE_REQUIRES_ARTIFACT

A runnable example must pass complete preflight as-is.

A template must identify typed substitution ports and cannot be emitted as a direct math.run argument until a host/server binds compatible values. #1031 and the typed request-local value-reference issue can supply those bindings.

4. Support canonical fixture generation for verifier examples

For generic replay verifiers, allow the owning domain to provide a small canonical producer input/result pair or a deterministic fixture builder that runs only during test/build preparation.

Do not infer a mathematically valid candidate from JSON Schema alone.

The resulting example should be stored as ordinary validated data and preflighted without invoking the checker during server startup.

5. Bind preflight identity to generated guidance and selected tools

• the generated guidance/direct-example manifest should include capability version, schema digest, and preflight-contract identity;
• [Superseded]: Improve concrete examples/contracts; do not add prepared-call tokens or dynamic selected tools #1031 selected tools should use the same preflight before adapter execution;
• [Design]: Validate parallel math.run batches before executing any member #896 batch validation should call the same preflight for every member;
• stale package/server examples should fall back to live exact inspection.

6. Keep provider readiness separate

An example may be contract-valid while the backend/provider is unavailable in the current process.

Report separately:

EXAMPLE_VALID
PROVIDER_UNAVAILABLE
POLICY_DENIED

Do not make provider availability part of mathematical request validity, and do not execute the provider merely to validate an example.

7. Test canonicalization, not only acceptance

If preflight normalizes accepted input, require the published example either:

• already be canonical; or
• explicitly show the canonical form and normalization behavior.

Prefer canonical examples so agents do not learn deprecated/noncanonical encodings.

8. Add plugin and catalog gates

Third-party adapter entrypoints should provide the same preflight contract. Reject plugins that advertise examples without a validation-equivalent preflight path.

Catalog health should report:

• capabilities with no runnable example;
• template-only examples;
• preflight failures;
• examples normalized during validation;
• example/schema/preflight digest drift.

Absence of an example can remain valid when the operation cannot supply a useful bounded fixture; it must not be confused with an invalid example.

Acceptance criteria

• Every installed capability uses one side-effect-free request preflight shared with runtime dispatch.
• Every RUNNABLE invocation example passes JSON Schema and the complete domain/Pydantic preflight.
• Installation fails closed when a built-in runnable example violates a cross-field or canonicalization invariant.
• Templates are typed and cannot be sent directly without binding required values/artifacts.
• Generic verifier examples include a domain-owned canonical candidate rather than a schema-inferred fake.
• Preflight writes no artifacts/state and invokes no mathematical provider backend.
• Provider unavailability and policy denial remain separate from request validity.
• Published examples are canonical or explicitly document deterministic normalization.
• Batch preflight, selected tools, generated guidance, and runtime dispatch use the same validation boundary.
• Plugin examples pass the same gate.
• Diagnostics use the unified recoverable taxonomy and never expose rejected values.

Regression fixtures

Include examples that are JSON-Schema-valid but should fail complete preflight:

1. matrix dimensions disagree;
2. polynomial variables differ across operands;
3. duplicate/noncanonical sparse terms;
4. typed artifact/schema mismatch;
5. verifier input/candidate labels disagree;
6. finite table fails closure/coverage;
7. digest does not bind canonical content;
8. budget fields are individually valid but jointly inconsistent;
9. wrong mode for an otherwise valid payload;
10. plugin-provided example with a hidden cross-field violation.

Also prove that preflight produces zero artifact/experiment/provider side effects.

Related work

• Expose schema-valid invocation examples for exact replay verifiers #941 and PR Clarify inline verifier invocation envelopes #929 own the immediate need for verifier invocation examples.
• [Superseded]: Improve concrete examples/contracts; do not add prepared-call tokens or dynamic selected tools #1031 owns schema-bound selected execution.
• [Design]: Validate parallel math.run batches before executing any member #896 owns atomic batch preflight.
• The generated-guidance issue owns versioned packaged examples.
• The canonical inline wire-type issue owns reusable primitive validation/canonicalization.
• The unified diagnostic-taxonomy issue owns recoverable preflight failures.

Non-goals

• Executing every mathematical example at server startup.
• Treating JSON Schema as unimportant; it remains the portable discovery/client contract.
• Inferring verifier candidates from structural schemas.
• Requiring every capability to have a static runnable example.
• Allowing validation preflight to write artifacts, mutate providers, or authorize checkers.
• Hiding provider unavailability by calling an unrelated fallback.

[morluto]

Current-main disposition (2026-08-15)

Evaluated against #1148 and the stateless MathTool declaration path.

Disposition: close as not planned.

Runnable examples should indeed pass the same concrete request model that math.run uses. The proposed universal preflight_request adapter protocol, canonical validated-request type, example-kind algebra, preflight identity, plugin gate, and shared preflight service are not needed.

The direct rule is:

• each MathTool owns one concrete request model / TypeAdapter;
• static examples are parsed by that model in ordinary tests/catalog construction;
• an example requiring runtime-produced data is documentation, not a falsely runnable template;
• operation-local preflight remains an ordinary pure function called after parsing only when the operation actually needs it;
• provider execution and checker authorization are never part of example validation.

Fix any failing example in its owning operation and add a focused regression. Do not introduce a generic preflight framework around every tool.