Skip to content

RELEASE_CONTRACT.md

Latest commit

 

History

History
229 lines (180 loc) · 11.8 KB

RELEASE_CONTRACT.md

File metadata and controls

229 lines (180 loc) · 11.8 KB
Raw

Release Contract

This document defines the stable Core Preview v0.5 release contract. A Loom release must keep these boundaries intact even while the implementation evolves. The versioned public CLI, inventory, manifest, dispatch, token, and capability surface is listed in Core Preview v0.5 Compatibility and Versioning.

V1 Task Contract

A manifest declares schema_version: 1 and expands each runnable identity into:

campaign_id + case_id + setting_id + run_id

The Hub assigns an attempt_no when that identity is leased. The task identity does not change across retries; the attempt does. Each attempt gets a fresh Runner work directory and one independently downloadable ZIP.

For repository tasks, defaults.phases declares ordered named phases. A case can override one by name through case.phases, or append a new named phase. Every phase may define command, args, cwd, env, timeout_seconds, continue_on_error, and artifact_paths.

Parameter precedence is explicit:

defaults.env < case.env < default phase.env < case phase.env < Loom runtime env

The Runner always injects the immutable values LOOM_TASK_ID, LOOM_ATTEMPT_NO, LOOM_WORKER_ID, LOOM_CAMPAIGN_ID, LOOM_CASE_ID, LOOM_RUN_ID, LOOM_SETTING_ID, LOOM_PHASE_NAME, and LOOM_PHASE_INDEX. Phase args are rendered from the case/default context before dispatch. See Loom Manifest for the full shape.

An optional extensions object lets integrations attach opaque JSON metadata without depending on Loom internals. Loom preserves the final payload.extensions value through Hub storage and into both task.json and worker-result.json (task_extensions); it never interprets that value for scheduling, retries, credentials, commands, or identity. Namespace keys merge atomically by documented layer precedence.

Inventory declares inventory_version: 1 and gives every worker an explicit initial_concurrency, hard max_concurrency, and concurrency_policy. The default fixed policy stays at its configured level except for an explicit resource-insufficient or rate-limit backoff; adaptive is opt-in.

Container Agent Contract

runner: "container" is an additive v1 task form. It has one versioned container object containing an immutable image descriptor, JSON config, explicit network_mode, secret environment-variable names, and an output allowlist. It cannot also supply a source checkout, repository phases, a non-empty payload.env, or arbitrary Docker flags.

The image descriptor is emitted by the public loom_container.py admit CLI, then published through loom_container.py publish. Hub verifies that exact descriptor and archive exist before dispatch. A Linux Runner verifies the image ID, linux/amd64 platform, required Loom labels, and runnable command before execution. It injects fresh file inputs and preserves only the declared output record and artifact paths in the attempt ZIP.

This is an admission and scheduling contract, not an Agent portability or quality guarantee. The image owner is responsible for making their Agent work on Linux and for its agent- or benchmark-specific behavior. See Docker Agent Protocol.

Scheduling And Direct Push

Hub is the only task-state owner. A Direct Runner never receives an unleased task and never owns a second queue.

direct-worker-api has two explicit dispatch modes:

  • pull: the long-lived Runner claims work from Hub. This is the default.
  • push: Hub leases one exact eligible task, then posts that lease to the Runner's authenticated /api/tasks/execute endpoint. The Runner reports start, result upload, completion/failure, and lease renewals back to Hub.

In both modes, long work renews its Hub lease while it runs. A failed delivery whose outcome is unknown remains leased until normal recovery, rather than being blindly dispatched twice.

For immutable Git task sources and admitted container image archives, cache locality is a soft preference after task priority and before FIFO tie-breaking among otherwise equal queued tasks. It never bypasses capability, retry, resource, placement, or concurrency checks. A container task additionally requires container-runtime-v1; a wildcard worker capability cannot stand in for a Runner's local runtime health check. push-task may omit worker_id to choose an eligible Direct Runner with the same preference; a supplied worker ID remains exact.

After every Direct Push execution, the Runner sends a completion heartbeat even when the task failed. That refreshes worker cache health and active-work facts without changing the Hub-owned task outcome.

Worker resource_capacity and task execution_profile are part of the same lease admission decision. Hub atomically reserves declared CPU, memory, disk, and accelerator values for leased and running tasks. shared placement may share a worker within those reservations; exclusive placement requires an otherwise idle worker. This is scheduler-level admission only, not a sandbox or OS-level resource guarantee.

Authentication And Network Boundary

Hub defaults to 127.0.0.1. Binding it outside loopback requires a bearer token from LOOM_HUB_TOKEN (or an explicitly named equivalent). A Direct Runner also defaults to 127.0.0.1; binding its control API outside loopback requires a separate bearer token from LOOM_RUNNER_TOKEN.

The Hub stores the name of a Direct Runner token environment variable in the host registry, never the token value. The Hub host resolves that value only when it delivers a push. Treat both endpoints as private control-plane services: use private addressing, firewall rules, or a TLS-terminating proxy when traffic leaves one trusted host. Loom does not provide TLS termination or identity management in Core Preview.

Result And Recovery Contract

Every repository result package includes at least:

  • task.json with the leased identity and attempt;
  • worker-result.json with process-level result metadata;
  • phase-results.json with one status record per executed phase; and
  • artifact-manifest.json with relative path, byte length, and SHA-256 for each declared artifact copied from the workspace.

When supplied, payload.extensions is retained in task.json and mirrored as task_extensions in worker-result.json.

For a repository task with a cacheable immutable Git source, worker-result.json additionally includes source_cache: cache key, hit/miss or repair state, approximate cached bytes added, materialization duration, and eviction or fallback facts. The cache key and canonical source identity are safe to query; source credentials and cache contents are not exported.

Result ZIPs exclude the source checkout. Retried attempts remain queryable by the same task ID and distinct attempt_no; a later clean attempt cannot erase a failure package.

Every container result package includes at least:

  • task.json with the leased identity and container descriptor;
  • worker-result.json with process result and container cache/admission facts;
  • container-summary.json with the exit and output-collection receipt;
  • container-record.json, copied from the required declared output path; and
  • artifact-manifest.json with byte length and SHA-256 for each copied output.

Container ZIPs exclude raw mounted config, task input, output staging, and workspace directories. Runner redacts exact declared secret values from command logs and refuses a record or declared artifact that contains one; secret_env names are permitted as contract metadata. Adapter owners remain responsible for not exporting transformed credentials. Container tasks currently do not start an Oracle child, so an execution that needs Oracle semantics must use a repository task until that separately versioned handoff is implemented.

Oracle, Trajectory, And Reward Contract

An optional v1 Oracle is a child task over one retained execution ZIP, not a field that overwrites the execution task. Hub records the child process state and the Oracle's semantic pass, fail, error, or inconclusive outcome independently. An oracle_error retry affects only the child task and continues to reference the exact parent ZIP by result ID, byte length, and SHA-256.

Trajectory capture is disabled unless the execution payload explicitly declares trajectory_export. When enabled, Runner enforces the configured size bound, redacts the structured document, records its SHA-256 receipt, and removes the raw trace before artifact collection and ZIP creation. Oracle-owned rewards are versioned structured output, not a Hub scheduling or training signal.

The public recovery selectors all_attempts, execution_clean, oracle_decided, and oracle_pass select retained ZIPs but never delete them. Each semantic selector includes both the Oracle package and the referenced execution package. The Oracle, Trajectory, And Reward Contract defines the exact fields, API queries, and information boundary.

Required Remote Release Check

The fixed AgentDojo release fixture is the minimum release gate. It uses two public cases and two run_id values, yielding four task identities. Each identity intentionally records:

  1. attempt 1: a retryable network_unavailable preflight failure; and
  2. attempt 2: one real AgentDojo invocation and declared artifact collection.

The gate passes only when all four tasks finish clean on attempt 2 and all eight attempt ZIPs are downloaded and hash-verified. A sanitized recovery summary is exported separately; raw ZIPs and model output are not committed.

Run it only on an already-provisioned remote host with the fixture's declared upstream execution environment and the two Loom bearer tokens available in that host's environment. The exact command and cleanup sequence are in Remote Validation.

Changes to immutable Git sources, Runner cache behavior, worker cache health, or cache-affine selection must additionally pass the source-cache remote gate. It is a four-task, two-Runner check for first fill, same-digest reuse, changed-digest refresh, corrupt-cache repair, automatic cache-affine Direct Push, and hash-verified result recovery. It requires no model provider and must run on a fresh remote host; its source-transfer metric records cache-fill work rather than public-internet bandwidth.

Changes to Oracle dispatch, parent-ZIP transfer, trajectory handling, reward storage, or semantic recovery selectors must additionally pass the Oracle remote gate. It verifies pass/fail/error/inconclusive, child-only retry, token-authenticated Hub/Runner calls, trace redaction, reward preservation, and SHA-256 recovery on a fresh remote host. The committed acceptance export is redacted; raw ZIPs, trace input, commands, and host identity remain outside Git.

Changes to the Docker Agent protocol, image admission, Hub archive delivery, Runner container execution, image-cache affinity, or container result recovery must additionally pass the Docker Agent and Flash remote gate. It runs on a fresh remote Linux host, makes one paid Flash provider request only after a verified image-archive delivery, then proves an offline second attempt reuses the same image with zero Hub transfer. The redacted export is evidence of the Loom control-plane contract only, not a real EnvTrustBench Agent or benchmark result.

Non-Goals

Loom does not create, resize, stop, delete, or price cloud resources. The remote smoke helper stops only the Hub and Runner processes it starts. The operator or an external infrastructure workflow must stop and delete the temporary host after evidence is copied out. See Loom Scope.