Skip to content

TASK_MANIFEST.md

Latest commit

 

History

History
347 lines (284 loc) · 11.9 KB

TASK_MANIFEST.md

File metadata and controls

347 lines (284 loc) · 11.9 KB
Raw

Loom Manifest

Loom does not infer benchmark work from a repository. A campaign author expands the work into explicit runnable records before dispatch.

Identity And Schema

Every JSON campaign must currently declare schema_version: 1 and every runnable record must resolve to:

  • campaign_id: one evaluation batch;
  • case_id: one benchmark problem or scenario;
  • setting_id: one model, defense, prompt, or environment slice; and
  • run_id: one concrete repetition of that case and setting.

Loom normalizes those fields into a stable ID:

{campaign_id}__{case_id}__{setting_id}__run-{run_id}

The same task ID survives retry. attempt_no is assigned by Hub and is the separate execution/recovery identity. A result package is therefore always addressable by (task_id, attempt_no).

V1 Repository Phases

runner: "repo" materializes a declared source checkout and executes ordered named phases. defaults.phases provides the common sequence; a case can update one by matching name, or append another phase with a new name.

{
  "schema_version": 1,
  "campaign_id": "example-release",
  "source": {
    "type": "git",
    "url": "https://github.com/example/project.git",
    "ref": "v1.2.3",
    "depth": 1
  },
  "defaults": {
    "runner": "repo",
    "required_capability": "linux",
    "env": {"LOG_LEVEL": "info"},
    "artifact_paths": ["reports/**"],
    "phases": [
      {
        "name": "prepare",
        "command": ["python3", "-m", "pip", "install", "."],
        "timeout_seconds": 600
      },
      {
        "name": "evaluate",
        "command": ["python3", "run_eval.py"],
        "args": ["--case", "{case_id}", "--run", "{run_id}"],
        "env": {"EVAL_MODE": "default"},
        "artifact_paths": ["reports/{case_id}/{run_id}/**"],
        "timeout_seconds": 1800
      },
      {
        "name": "collect",
        "command": "python3 collect.py",
        "continue_on_error": false
      }
    ]
  },
  "cases": [
    {
      "case_id": "case-a",
      "setting_id": "baseline",
      "run_id": "001",
      "phases": [
        {"name": "evaluate", "args": ["--case", "{case_id}", "--run", "{run_id}", "--seed", "7"]}
      ]
    }
  ]
}

Allowed phase fields are name, command, args, cwd, env, timeout_seconds, continue_on_error, and artifact_paths. command accepts a shell string or an executable argument list; args are appended safely for list commands and shell-quoted for string commands. cwd must be a relative path inside the materialized workspace.

Legacy commands remain supported when phases is absent. New manifests should use phases so the prepare, evaluate, and collection contract is visible to the controller, operator, and result consumer.

V1 Container Agent

runner: "container" executes an adapter-owned immutable Linux image instead of a repository checkout. Its required container object is selected atomically at campaign/default/case payload precedence, and container.config is rendered with the same {case_id}, {run_id}, and related context variables used by phases.

The image descriptor must have been produced by tools/loom_container.py admit and published to Hub before dispatch. It includes an archive ID, byte length, SHA-256, Docker image ID, linux/amd64 platform, agent ID, and integration. Do not hand-write an abbreviated descriptor into a runnable manifest.

{
  "defaults": {
    "runner": "container",
    "container": {
      "schema_version": 1,
      "image": {
        "schema_version": 1,
        "archive_id": "image-sha256-...",
        "archive_sha256": "...",
        "archive_bytes": 123456,
        "image_reference": "loom/envtrustbench-opencode:linux-amd64",
        "image_id": "sha256:...",
        "platform": "linux/amd64",
        "agent_id": "envtrustbench/opencode",
        "integration": "envtrustbench"
      },
      "network_mode": "bridge",
      "secret_env": ["MODEL_API_KEY"],
      "config": {"scenario_id": "{case_id}", "repeat": "{run_id}"},
      "output": {"record_path": "record.json", "artifact_paths": ["summary.json"]}
    }
  }
}

config must be a JSON object without inline credentials. secret_env lists host environment variable names; Runner injects their values only into the container process. The image receives a read-only config.json and task.json under /loom/input, then must write a JSON record.json under /loom/output. Use container.output.artifact_paths for the output allowlist. Container tasks do not accept a Git source, ordered phases, arbitrary Docker flags, or non-empty payload.env; put non-secret settings in config and rebuild the immutable image for runtime changes.

See Docker Agent Protocol for required labels, archive publication, resource-derived Docker limits, cache affinity, and the EnvTrustBench adapter boundary.

Immutable Git Sources

When a Git source is pinned to a complete commit with resolved_commit, commit, or a full object ID in ref, normalization adds an immutable payload.source_descriptor. It contains a canonical credential-free URL, commit, and derived cache key. Mutable branches and tags intentionally do not receive a descriptor, so they cannot use stale source cache entries.

The descriptor is generated by Loom; do not hand-author it in a campaign. Raw dispatch clients may send one only when it exactly matches the accompanying source. Cache Affinity defines cache configuration, result facts, and scheduling behavior.

Execution Profile

execution_profile declares scheduler placement and a resource reservation. It may appear in defaults, a case/run record, or payload; case values merge over defaults by resource field.

{
  "execution_profile": {
    "placement": "shared",
    "resources": {
      "cpu_millis": 750,
      "memory_mb": 1536,
      "disk_mb": 2048,
      "gpu_count": 0
    }
  }
}

placement is shared by default. exclusive reserves a worker until the attempt leaves its active lease state. Resource fields are scheduler admission requests, not OS-level limits or a claim of security isolation. See Resource Admission for inventory capacity, GPU type constraints, and inspection commands.

Extensions

extensions is an optional opaque JSON object for metadata owned by an integration, dataset, or downstream result consumer. Loom preserves it from manifest normalization through task dispatch and result recovery, but does not use it for scheduling, retries, authentication, command construction, or task identity. Do not place credentials in it.

Use stable namespace-like keys such as org.example.dataset or com.acme.oracle to avoid collisions with another integration. Each value must be JSON-compatible; Loom does not interpolate templates inside it.

{
  "extensions": {
    "org.example.dataset": {
      "dataset_revision": "2026-07",
      "trajectory_export": "redacted"
    },
    "com.acme.oracle": {
      "reward_profile": "safety-v3"
    }
  }
}

The field may be declared at campaign, defaults, or case/run level. It is also accepted in defaults.payload and case.payload for generated manifests. Within a normalized task, precedence is:

campaign.extensions < defaults.extensions < defaults.payload.extensions
< case.extensions < case.payload.extensions

Extensions merge only by their top-level key: a later com.acme.oracle replaces the complete earlier com.acme.oracle value. This keeps nested merge semantics owned by the extension author. For a raw dispatch payload, the corresponding order is dispatch.extensions, dispatch.payload.extensions, task.extensions, then task.payload.extensions.

The normalized task carries the final value as payload.extensions. Every result ZIP preserves it in task.json and mirrors it in worker-result.json as task_extensions, so an exporter can consume it without reinterpreting an arbitrary manifest.

Oracle And Trajectory Contracts

An execution task may optionally declare oracle and trajectory_export at campaign, defaults, case/run, defaults.payload, or case.payload level. They are public versioned contracts, not extension metadata. The selected object is atomic: later layers replace the full earlier object rather than merging nested fields. This keeps an Oracle's execution code, resource profile, retry policy, and semantic version internally consistent.

oracle queues one separately leased child after a retained execution result; trajectory_export asks the Runner to redact a structured trace before it builds the attempt ZIP. Both contracts are opt-in and require schema_version: 1. See Oracle, Trajectory, And Reward for the exact schema, runtime variables, result format, and export selectors.

Parameters And Runtime Injection

The normalizer expands {campaign_id}, {case_id}, {setting_id}, {run_id} and any default/case field in phase strings, args, paths, and environment values. Environment precedence is:

defaults.env < case.env < default phase.env < case phase.env < Loom runtime env

The final runtime values cannot be overwritten by a manifest:

LOOM_TASK_ID
LOOM_ATTEMPT_NO
LOOM_WORKER_ID
LOOM_CAMPAIGN_ID
LOOM_CASE_ID
LOOM_RUN_ID
LOOM_SETTING_ID
LOOM_PHASE_NAME
LOOM_PHASE_INDEX
LOOM_TASK_KIND

Use args for phase-specific main-script parameters and env for compatible tool configuration. Do not put credentials in a manifest, command URL, or task ID. Keep credentials in the remote worker environment.

For runner: "container", use container.config rather than env; the protocol passes only the declared container.secret_env names through to Docker and writes the fixed Loom runtime fields into /loom/input/task.json.

Retries And Expected Results

retry_policy belongs to an execution identity, not to a cloud host:

  • max_attempts: total attempts including the first;
  • retry_categories: Hub issue categories that permit automatic retry; and
  • different_worker: require the retry to land on another capable active worker, otherwise retain the failed task rather than waiting forever.

Normally retry only transient categories such as network_unavailable. Do not blindly retry auth_failed, token_balance_insufficient, or resource errors.

expected is an optional regression contract:

  • state: required final task state;
  • attempt_no: required final attempt;
  • min_result_count: minimum retained result ZIPs; and
  • min_distinct_workers: minimum workers represented by those ZIPs.

When a repository attempt starts, its working directory is fresh. Its result ZIP contains task.json, worker-result.json, phase-results.json, artifact-manifest.json, command logs, and explicit declared artifacts. The source checkout is excluded.

Selection, Normalize, And Dispatch

Normalize all tasks or narrow the selected identity dimensions before dispatch:

python3 tools/loom_manifest.py campaign.json \
  --case-id case-a \
  --run-id 001 \
  --setting-id baseline \
  --operator my-team \
  --output campaign.dispatch.json

The three selectors are repeatable. An empty selection is an error rather than a quiet no-op.

For an authenticated Hub:

export LOOM_HUB_TOKEN='...'

python3 tools/loom_hub.py dispatch-spec \
  --controller http://CONTROL_HOST:8765 \
  campaign.dispatch.json

curl -sS \
  -H "Authorization: Bearer $LOOM_HUB_TOKEN" \
  'http://CONTROL_HOST:8765/api/tasks?case_id=case-a&setting_id=baseline&run_id=001'

Retry one identity without mutating neighboring cases/runs:

python3 tools/loom_hub.py retry-task \
  --controller http://CONTROL_HOST:8765 \
  --task-id example-release__case-a__baseline__run-001

Earlier failure packages remain downloadable after a later clean retry. See Release Contract for Direct Runner push, authentication, and release-gate requirements.

JSONL

JSONL is available for large flat batches. Each line must carry the same explicit campaign_id and its own case/run identity; top-level defaults are not available in JSONL mode.