[Architecture]: Add typed request-local value references for capability composition without manual JSON copying

[morluto]

Priority

P1/P2 architecture issue: even after an agent selects the correct capabilities and schemas, composing them often requires manually copying or reconstructing large typed JSON values between math.run calls. This creates transcription, ordering, provenance, and verification-binding failures.

Area

Capability composition, inline values, producer/consumer handoff, verifier candidates, result references, schema/semantics binding, request/session lifetime, artifacts.

Summary

Jacobian currently has two primary value-transfer boundaries:

1. inline structured JSON inside a capability payload/result;
2. durable typed artifacts identified by artifact:// URIs and schema/semantics contracts.

There is no common typed reference for an ordinary bounded value already produced in the current request/session but not intentionally persisted as a durable artifact.

As a result, the agent often must:

• locate the relevant field in one arbitrary output object;
• copy it into another capability's nested payload;
• duplicate interpretation metadata such as variable/vertex order;
• construct a derived operand such as a transpose;
• preserve exact-number encodings and wrapper levels;
• avoid accidentally changing one entry;
• convince a verifier that the copied candidate is the exact producer result.

The runtime validates each call safely, but the composition boundary is model-authored serialization.

Audit baseline: current main as inspected on 2026-08-10.

Concrete evidence

Matrix operand duplication and #957

A held-out incidence-matrix task needed the natural operation:

A A^T

The installed matrix.multiply.compute contract required the full left and right matrices explicitly. The weak model repeatedly omitted or misplaced the derived right operand, made four invalid calls, and then calculated manually.

PR #957 evaluated a derived_operand treatment. It did not pass its keep gate: valid derived calls appeared in only one lineage and formed D D^T where the task needed D^T D.

The correct disposition was not to ship a generic transform language, standalone transpose capability, or artifactize ordinary values. The study explicitly leaves open a narrower target-local typed reference that binds operand position/order directly.

Verifier candidate copying

PR #929 and issue #941 show agents misconstructing the common producer/verifier handoff:

input and candidate as siblings inside payload

#1031 addresses schema-bound invocation so the host can constrain the wrapper shape. It does not remove the need to copy the exact producer candidate into that shape.

Large inline result with no artifact reference

A Jacobian-syzygy trajectory recorded under #28/#195 returned a complete result of roughly 92.7 KB inline with no result/artifact URI. The related verifier was discoverable, but the model had to reconstruct the large candidate manually and two attempts failed closed.

Native ResourceLink support cannot solve a value for which the producer deliberately returned no durable resource.

Positional semantic detachment

#930/#986 show that copying a distance matrix separately from its authoritative labels/order can turn a correct producer output into a semantically wrong certificate.

Artifact-only values can also be unusable

PR #992 shows the opposite extreme: the complete flat lattice was durable and independently verified, but the host/model could not read a useful value and reconstructed it manually. #1033 owns the bounded primary-result projection. A composition reference must work with either an inline canonical source or a durable artifact without confusing the two.

Current contract boundary

CapabilityDescriptor can advertise:

accepted_input_kinds
accepted_artifact_types
produced_artifact_types

The coarse input kinds include structured requests and typed artifacts. A typed artifact is appropriate when the value needs durable identity, retrieval, replay, evidence binding, or size-separated transport.

CapabilityRequest.input and CapabilityResult.output are otherwise ordinary dictionaries. There is no typed field reference or ephemeral value identity for composition.

Relevant paths:

• src/jacobian/contracts/capabilities.py
• src/jacobian/adapters/mcp/tools.py
• src/jacobian/adapters/mcp/tooling.py
• domain request/result contracts
• artifact/schema/semantics registries

Root cause

The architecture conflates value transport with value persistence.

A value produced by one operation may need exact type/provenance binding for the next operation without needing a durable globally addressable artifact.

The current choices are:

copy JSON manually
or
materialize a durable artifact

Manual copying is fragile. Universal artifactization adds storage, lifecycle, schema-registration, and discovery overhead to ordinary local composition.

The missing abstraction is a bounded, typed, tenant/session-scoped reference to an exact canonical value already held by Jacobian or the host.

Proposed architecture

1. Introduce a typed value-reference contract

A possible model:

{
  "value_ref_version": "1",
  "value_uri": "value://...",
  "schema_uri": "artifact://sha256/...",
  "semantics_uri": "artifact://sha256/...",
  "value_digest": "sha256:...",
  "source_invocation_uri": "invocation://...",
  "source_pointer": "/output/result",
  "lifetime": "REQUEST | SESSION | DURABLE_ARTIFACT",
  "tenant_binding": "opaque"
}

Exact fields can differ. The reference must bind:

• exact canonical value bytes;
• structural schema and mathematical semantics;
• producing capability/version/invocation;
• interpretation-critical metadata;
• tenant and lifetime;
• whether it is merely an inline/session value or a durable artifact.

The URI itself is not evidence and grants no authority.

2. Let domains declare referenceable output ports

Do not expose arbitrary JSON-pointer access to every output field.

A capability descriptor/family should identify domain-owned output ports such as:

matrix
labelled_graph
polynomial
candidate_result
finite_table
verification_record

Each port binds a schema/semantics contract and any required adjacent interpretation fields.

For a distance matrix, the referenceable value must include row/column labels/order atomically rather than referencing the numeric matrix alone.

3. Let consumers declare accepted value ports

A request field may accept:

inline exact value
or
compatible typed value reference

Compatibility is checked by the runtime against declared schema/semantics/relationship metadata, not inferred by the model from matching JSON field names.

This should extend #1018's compatibility graph beyond durable artifacts while preserving its typed ownership rules.

4. Keep transformations explicit and domain-owned

Do not reintroduce #957's failed generic transform/expression language.

A derived reference should use a small domain-owned transform where needed, for example:

{
  "source": {"value_uri": "value://..."},
  "transform": {"kind": "MATRIX_TRANSPOSE"}
}

Only transforms declared by the target/domain contract are valid. The transform binds operand position, dimensions, labels, and resulting schema.

For D^T D, the target request should make left/right source and transform explicit so D D^T is not a schema-valid accidental substitution.

No generic JSONPath, arbitrary expression evaluator, or model-authored code belongs in this boundary.

5. Prefer host/server insertion over model-managed IDs

Where supported, the host should present compatible produced values as selectable typed inputs or automatically preserve the exact reference when constructing the next selected tool from #1031.

The model may reason about “use the previous matrix as the right operand,” but should not have to transcribe opaque digests or long URIs manually.

Generic math.run can retain an explicit reference envelope as a compatibility path.

6. Make lifetime finite and explicit

Suggested states:

REQUEST
SESSION
DURABLE_ARTIFACT

Request/session values:

• are not enumerable globally;
• expire deterministically;
• cannot be resolved across tenants;
• are bounded by count/bytes;
• are garbage-collected independently of durable artifacts;
• fail with a typed VALUE_REFERENCE_EXPIRED diagnostic and a safe reproduction path when possible.

Promotion to a durable artifact is an explicit operation/policy decision, not automatic.

7. Preserve verification boundaries

A reference to a producer output does not transfer assurance to a derived value.

A verifier must bind:

• the exact dereferenced value;
• producer input/result lineage where required;
• semantics/schema;
• candidate/claim scope;
• authorized checker identity.

A transform creates a new value identity and, unless the checker covers the transform, leaves an explicit obligation.

8. Integrate with primary-result and invocation identity

• [Superseded]: Do not add a primary-projection framework; return complete results or a small domain-specific summary #1033 should identify which bounded canonical output is the primary/referenceable value.
• the cancellation-recovery issue should preserve produced value references when an invocation completes late;
• [Superseded]: Improve concrete examples/contracts; do not add prepared-call tokens or dynamic selected tools #1031 should let a selected consumer tool accept compatible references structurally;
• [Design]: Split mutating controls from pure math.run so MCP annotations are truthful #1034 effect metadata determines whether resolving/transforming/promoting a value is read-only or additive.

9. Bound context and storage

Reference creation must not copy a 100 KB value into every result channel.

Return a small typed port/reference plus the bounded primary preview. Keep canonical value bytes once in the request/session store or existing artifact store.

Acceptance criteria

• A producer can declare one or more typed referenceable output ports without making every output field globally addressable.
• A consumer can accept a compatible inline value or typed reference under one validation-equivalent contract.
• Schema, semantics, source invocation/version, canonical digest, tenant, and lifetime are bound in every reference.
• Matrix D^T D can be expressed without copying D and without allowing operand-order ambiguity.
• The Clarify inline verifier invocation envelopes #929/Expose schema-valid invocation examples for exact replay verifiers #941 producer-verifier cases can pass the exact candidate by reference without manual JSON reconstruction.
• A large inline syzygy result can be consumed by its verifier without becoming a permanent artifact solely for handoff.
• A labelled distance matrix cannot be referenced separately from its labels/order.
• Expired, cross-tenant, wrong-schema, wrong-semantics, stale-version, and tampered references fail closed with typed diagnostics.
• Transform references are finite, domain-owned, and bind dimensions/ordering; no generic expression language is introduced.
• Assurance never transfers automatically across a reference or transform.
• Request/session reference counts and bytes are bounded and observable.
• Explicit promotion to durable artifact preserves provenance and complete artifact-contract policy.

Suggested evaluation

Use frozen tasks under four arms:

1. manual inline copying;
2. durable artifact handoff where available;
3. invocation example/schema-bound call without value reference;
4. typed request/session value reference.

Fixtures:

• D^T D versus D D^T;
• nested verifier input/candidate;
• large syzygy candidate;
• labelled distance matrix;
• projective-flat materialization;
• small scalar where a reference should be unnecessary.

Measure:

• first valid consumer call;
• transcription/order errors;
• bytes copied into model context;
• resource/value dereference success;
• task correctness and assurance calibration;
• storage writes;
• expired/cross-tenant recovery;
• elapsed time and retries.

Related work

• [Feature]: Make capability contracts easier for agents to discover and compose #28 is the broad composition/ergonomics umbrella.
• PR eval: record matrix operand composition study #957 is important negative evidence against a generic derived-operand language; retain it as a regression.
• Restack typed capability compatibility metadata after reasoning-workflow removal #1018 owns typed durable artifact producer/consumer metadata.
• [Superseded]: Improve concrete examples/contracts; do not add prepared-call tokens or dynamic selected tools #1031 owns schema-bound selected execution.
• [Superseded]: Do not add a primary-projection framework; return complete results or a small domain-specific summary #1033 owns bounded primary-result projection and source binding.
• [Evaluation]: numeric-looking graph labels can misbind distance-matrix rows #930/Bind graph distance-matrix entries to vertex labels #986 own the labelled distance-matrix reproduction.
• Clarify inline verifier invocation envelopes #929/Expose schema-valid invocation examples for exact replay verifiers #941 own verifier envelope examples.
• [Architecture]: Persist complete artifact contracts, not only JSON Schema shape #1019 owns durable artifact contract persistence; request/session values must not bypass it when promoted.

Non-goals

• Persisting every intermediate value as a durable artifact.
• Exposing arbitrary JSON pointers or a generic expression language to the model.
• Hiding mathematical transforms inside implicit server magic.
• Treating a value reference as verification evidence or checker authorization.
• Sharing mutable values across tenants.
• Replacing domain-owned request/result schemas with a universal object model.

[morluto]

Value references must resolve declared ports, not arbitrary output pointers

#1208 should be a prerequisite. A request/session reference should identify one producer-declared output port and one registered mathematical value type; consumers accept it through a compatible input port.

Prefer:

source_invocation + source_port + value_type/domain/axes/canonicalization digest

over arbitrary /output/... JSON pointers. The runtime can still use internal pointers generated from the typed contract, but models/clients should not author them.

This also narrows transforms: a transform is valid only when declared between source and target port types. No generic JSONPath/expression language and no untyped value store.

[morluto]

Superseded by #1483, revised #1074, #1490, and #1492.

The original issue correctly diagnosed that model-authored copying of large exact values is fragile. The proposed value:// request/session store is no longer the target because it creates another opaque server-local identity and hidden mutable state.

The replacement composition rules are:

Python -> pass typed ResultT directly
MCP/process -> read/pass the complete canonical typed document
large immediate value -> bounded call-scoped resource, then complete document bytes
caller workspace -> resolve its own reference to a complete document before invoking Jacobian

Declared input/output value types and interpretation-critical axes/labels remain useful. They should constrain direct typed/document composition, not index a core-local value table. Closing as not planned.