TODO.md

TODO

v0.1.0 Release Baseline

• Freeze the product release as v0.1.0 Core Preview while keeping the independently versioned inventory, manifest, dispatch, and API contracts at v1.

Configuration Circuit Breaker

• Add an explicit configuration-level circuit breaker, scoped at minimum to campaign_id + setting_id. An authenticated operator or evaluation agent that attributes a recovered artifact anomaly to an error class must be able to trip it with an error class, reason, and task/result evidence reference.
• When open, block every new Pull claim and Direct Push lease for matching execution, retry, and Oracle work while retaining queued tasks and evidence. Do not silently cancel already leased or running attempts; cancellation must remain an explicit separate operator action.
• Persist breaker state and an append-only audit trail. Provide authenticated inspection and resume operations so the affected configuration can be unblocked only with an operator, reason, and recorded recovery event.
• Add remote-only coverage for configuration isolation, Pull and Direct Push blocking, retry suppression, in-flight attempt preservation, evidence validation, and explicit resume.

Next: Task-Graph Splitting And Stage Scheduling

• Promote runner-local phases into an opt-in, versioned task-graph contract. A campaign must be able to expand one case/run into deterministic independently schedulable stage tasks with explicit dependencies, while legacy ordered phases remain one execution attempt for compatibility.
• Give every stage its own controller-owned identity, lease, attempt history, resource profile, capability requirement, priority, timeout, and retry policy. A failed evaluate stage must not rerun a completed expensive prepare stage, and cancelling or retrying one stage must not mutate its siblings.
• Define a content-addressed stage handoff contract: downstream stages request named upstream artifacts by declared SHA-256 and materialize only those validated inputs. Do not pass an opaque writable workspace between workers or silently infer dependencies from paths.
• Make stage-specific command arguments, environment, working directory, artifacts, and Loom runtime metadata explicit in the manifest. Preserve the existing case/run/attempt injection and add immutable stage and dependency identifiers so one main script can distinguish preparation, evaluation, and collection safely.
• Schedule only dependency-ready stages through the existing fairness, priority, capability, resource-admission, concurrency, and cache-affinity rules. A cache hit remains a preference; dependency readiness and hard capacity limits remain requirements.
• Add remote-only release coverage for a multi-worker stage graph: independent case/run selection, upstream artifact handoff and hash failure, stage-only retry/cancel, blocked-dependency recovery, and result ZIP retention. Keep the fixed AgentDojo 2-case x 2-run x 2-attempt fixture as the execution-level regression, not evidence that stage splitting works.

Release Contract And AgentDojo Example

• Add a versioned phases manifest contract that preserves the current commands format. A phase needs a name, command, optional args, environment, working directory, timeout, and declared artifacts. Define and document the precedence as campaign defaults, case/run overrides, phase overrides, then immutable Loom runtime metadata.
• Inject campaign_id, case_id, setting_id, run_id, attempt_no, worker ID, and phase identity into the phase process. Emit a structured phase report and an artifact manifest with paths, byte counts, and hashes.
• Keep case and run independently selectable at dispatch time. Keep attempt controller-owned: a retry creates the next attempt for the same task/recovery identity rather than accepting an arbitrary attempt number.
• Make remote Hub and direct Runner APIs secure by default: loopback-only bind defaults, bearer-token configuration through environment variables, and authenticated registration, dispatch, result upload, and direct-worker calls.
• Add remote-only automated coverage for manifest normalization, per-phase injection, case/run filtering, retained retry attempts, result recovery, and authenticated HTTP APIs. Run deterministic coverage on an existing remote host; run the model-backed smoke test as a remote release job rather than on every commit.
• Add a minimal, pinned AgentDojo campaign with two independently selectable case/run records and a retry demonstration. Run it on a new inexpensive remote host, collect and redact the returned result package, and publish the recovered task.json, phase report, score/log summary, and artifact manifest under examples/agentdojo/.
• Keep cloud resource creation out of Loom's supported scope. The remote validation recipe must explicitly include host shutdown/cleanup as its final step.

Next: Content-Addressed Cache And Cache-Affinity Scheduling

• Add an immutable Git source descriptor to the manifest and dispatch contract using a canonical URL plus resolved commit. Generic input bundles still need a separate URI, byte-length, and SHA-256 contract.
• Give every Runner a bounded, lock-protected local Git source cache. A hit creates a fresh writable worktree per attempt; fill, verification failure, eviction, and corruption recovery are observable. Generic input caching is still pending.
• Make cache locality a soft scheduling preference for both Pull and Direct Push. Capability, resource admission, priority, and fairness remain hard constraints; a cache miss never blocks an otherwise eligible task.
• Report cache key, hit/miss, transferred bytes, materialization time, and eviction facts in Runner results and Hub queries without exposing source credentials or turning Hub into a required blob store.
• Add a fixed remote source-cache release gate covering same-digest reuse, changed-digest refresh, corrupt-cache recovery, cache-affine Direct Push, SHA-256-verified ZIP recovery, and an explicit Direct Runner cache-health heartbeat after push completion.
• Benchmark external repository network, wall-clock, disk, and queueing impact across operator-supplied hosts. The fixed release gate records deterministic local-fixture cache-fill metrics, not public-network claims.

Docker Agent Admission And Reuse

• Define a public Dockerfile/image admission protocol for adapter-owned linux/amd64 Agent images: fixed labels, immutable archive descriptor, file-configured case/run input, declared output record, and secret-name-only forwarding. Loom does not own an Agent's Windows-to-Linux port or benchmark behavior.
• Publish admitted image tars to Hub with byte-length/SHA-256 validation, require a matching published descriptor before dispatch, and verify image ID, platform, labels, and command again on the Runner before execution.
• Add bounded verified image-archive caching and soft cache-affine Pull and automatic Direct Push selection. A container task still requires a Runner's independently observed Docker runtime and cannot bypass priority, capacity, placement, retry, or concurrency admission.
• Add the EnvTrustBench adapter catalog and Dockerfile template for Codex, Claude Code, OpenCode, OpenClaw, and Gemini CLI. Catalog membership is an integration target, not a claim of current Linux portability.
• Run the fresh-host remote Docker/Flash release gate, recover only its redacted two-attempt acceptance export, and explicitly delete/verify removal of the test resource. The fixture uses one provider call and one offline cache-reuse attempt; it is not a benchmark validation.

After Cache: Oracle, Trajectory, And Reward

• Add a separately schedulable oracle contract. Hub should queue it from an execution result reference, with its own capability, resource profile, timeout, retry policy, and attempt history.
• Keep execution and Oracle outcomes independent: execution success/error/ timeout must not be confused with Oracle pass, fail, error, or inconclusive. Oracle retries must not rerun a successful expensive Agent attempt.
• Add explicit recovery/export selectors such as all_attempts, execution_clean, oracle_decided, and oracle_pass. Retain raw attempt ZIPs by default; selectors control export, never silent evidence deletion.
• Define an opt-in, versioned trajectory export contract for agent messages, tool calls, observations, timings, and artifact references. Include redaction and size policies; raw trajectory capture must remain disabled by default.
• Define a versioned reward contract owned by Oracle output, supporting a scalar reward, optional named components, score metadata, Oracle version, and evidence references. Loom exports this data but does not become an RL trainer.
• Add remote release coverage for Oracle pass/fail/error/inconclusive, trajectory redaction, reward integrity, and export selection across retries.

Resource Multiplexing And Local Mode

• Make shared-host resource multiplexing an explicit Loom capability with task resource reservations and shared / exclusive placement.
• Document that the scheduler improves use of existing capacity without claiming an advantage for CPU-bound or strict-isolation workloads.
• Benchmark throughput, CPU and memory utilization, queueing delay, failure interference, and cost per completed task at different concurrency levels.
• Add operator-owned isolation adapters for direct host execution, separate processes, containers, or stronger external sandboxes without making Loom a cloud lifecycle tool.
• Add a one-command local mode that starts an embedded Hub and localhost Runner automatically while preserving the same scheduling, lease, retry, and result-recovery semantics as multi-host mode.