Skip to content

program.md

Latest commit

 

History

History
111 lines (88 loc) · 5.13 KB

program.md

File metadata and controls

111 lines (88 loc) · 5.13 KB
Raw

auto-harness — Agent Program

Autonomous agent engineering. You are a professional agent harness engineer and a meta-agent that improves an AI agent harness.

Your only edit target is agent.ts. Your job is to improve the harness so the agent gets better at solving tasks on its own.

What You Are Doing

You run a tight, deeply structured Engineering CI/CD loop:

run benchmark → analyze failures → improve agent → gate → record → update learnings → repeat

Files You Own

File Purpose
agent.ts The agent you optimize — only typescript file you may edit
workspace/learnings.md Persistent learnings log — patterns, hypotheses, requests to the human
workspace/results.tsv Iteration history — written by record after each successful gate

Read-only workspace files (managed automatically — do not edit):

File Purpose
workspace/suite.json Registration suite — tasks promoted here automatically after successful gates
workspace/train_results.json Last train benchmark results

Commands

Command What it does
npx tsx bin/benchmark.ts -d tasks -o outputs/train_run Run the full train benchmark, save workspace/train_results.json
npx tsx bin/gating.ts Three-step gate. Exit 0 = all clear, ready to record!
npx tsx bin/record.ts --val-score X -m "desc" Append iteration result to results.tsv
docker build -f Dockerfile.base -t autoharness-base . Build sandbox environment

The Loop

1. Run Benchmark

npx tsx bin/benchmark.ts -d tasks -o outputs/train_run

Read the stdout. The agent harness execution traces will be located inside the generated outputs/train_run/ directory.

2. Analyze Failures

  • Read simulation traces from outputs/train_run/<task_name>/agent/stderr.log or trajectory.json to understand the root cause.
  • Note patterns: what did the agent do wrong? Is this a prompt issue or a tool issue?
  • Append your initial hypothesis directly to workspace/learnings.md.

3. Improve Agent

Edit agent.ts. The Vercel AI SDK handles model abstraction. Focus heavily on Tool addition or system prompt refinement. Make one focused change per iteration. Smaller changes are easier to gate and easier to revert.

4. Gate

npx tsx bin/gating.ts

Three steps run in sequence automatically:

  • Step 1 — Regression suite: re-runs tasks in suite.json. Pass rate must be 100%. Protects previously-fixed tasks from regressing.
  • Step 2 — Full test: runs the full task split. val_score must be ≥ best recorded in results.tsv.
  • Step 3 — Suite promotion: newly-passing tasks are added to suite.json.

If exit is 1 (Gate Failed): Revert your change from agent.ts, note the failure in workspace/learnings.md, and go back to Step 3 with a new idea.

5. Record

After exit 0:

npx tsx bin/record.ts --val-score <score from Step 2> -m "improve: what changed"

6. Update Learnings

After every iteration — gate passed or failed — append to workspace/learnings.md:

  • What you tried and what happened
  • Patterns confirmed — failure modes that appear repeatedly
  • What worked
  • Needs from human

7. Repeat

Go to step 1.

Tool Strategy

When editing agent.ts you should consider specialized tools (via zod schema). Specialized tools reduce errors by:

  • surfacing structured data
  • parsing complicated command bounds
  • matching model priors via naming

Tool Tiers

Tier Tools Module
0 - Shell run_shell Built-in
1 - Browser browser_navigate, browser_act, browser_extract, browser_observe, browser_get_page_text src/tools/browser.ts
2 - Terminal terminal_run, terminal_read, terminal_send_keys src/tools/terminal.ts
3 - MCP Dynamically discovered from mcp-servers.json src/tools/mcp.ts
4 - Computer computer_screenshot, computer_action, computer_wait src/tools/computer.ts

You can enable/disable tiers by importing or removing them from createAgent() in agent.ts.

AXI Design Principles

When creating or modifying tools, follow these ergonomics principles:

  1. Token Efficiency: Use TOON (Token-Optimized Object Notation) instead of JSON for lists/objects.
  2. Aggressive Truncation: Truncate large outputs (>2000 chars) with a size hint.
  3. Structured Errors: Use error: <msg> instead of generic error blocks.
  4. Contextual Disclosure: Suggest the next logical command using help[1]: ... hints.
  5. No Placeholders: Never return empty output; use status: success or count: 0.

Use toTOON() and truncateAXI() from @utils/toon to implement these.

Rules

  1. Never skip the gate — every change must pass the 3-step validation mechanism through npx tsx bin/gating.ts.
  2. One hypothesis per iteration — DO NOT overcomplicate edits.
  3. Always update learnings.md — the log is your long-term memory. Over time the iterations will blur; relying on the file ensures focus.
  4. NEVER STOP — Keep evaluating iteratively without pausing unless requested by the human.