-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add Natural-Language Agent Harness (NLAH) contract system #28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
frankxai
wants to merge
3
commits into
main
Choose a base branch
from
claude/meta-harness-analysis-3dYH4
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # NLAH execution artifacts are project-local and ephemeral. | ||
| # Only the .gitignore itself is committed. | ||
| artifacts/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| # ACOS Runtime Charter v1.0 | ||
|
|
||
| This charter defines how the ACOS runtime interprets and enforces Natural-Language Agent Harness (NLAH) contracts. It is loaded at session start and governs all contracted skill execution. | ||
|
|
||
| This document is itself an NLAH — a natural-language specification interpreted by the in-loop LLM. | ||
|
|
||
| --- | ||
|
|
||
| ## 1. Contract Interpretation | ||
|
|
||
| When a skill with a `contract:` block in its SKILL.md frontmatter is activated: | ||
|
|
||
| 1. **Parse the contract** from YAML frontmatter | ||
| 2. **Initialize tracking** — set tool-call counter to 0, file-edit counter to 0, child-agent counter to 0 | ||
| 3. **Create artifact directory** at the resolved `artifacts.state-dir` path | ||
| 4. **Write contract snapshot** to `contract-snapshot.json` in the artifact directory | ||
| 5. **Begin execution** — the skill's markdown body is the harness logic | ||
|
|
||
| ### Variable Resolution | ||
|
|
||
| Template variables in contract fields are resolved at activation: | ||
| - `${skill-name}` → the `name` field from frontmatter | ||
| - `${session-id}` → a unique identifier for this execution (timestamp-based) | ||
|
|
||
| ## 2. Budget Enforcement | ||
|
|
||
| Budget enforcement operates through awareness, not hard blocks. The in-loop LLM reads budget state and adjusts behavior accordingly. | ||
|
|
||
| ### Budget Tracking | ||
|
|
||
| After each tool call, update counters: | ||
| - `tool_calls` += 1 for every tool invocation | ||
| - `file_edits` += 1 for every Write or Edit call | ||
| - `child_agents` += 1 for every Agent or Task call | ||
|
|
||
| ### Budget Signals | ||
|
|
||
| | Usage Level | Signal | Expected Behavior | | ||
| |---|---|---| | ||
| | 0-70% of any limit | None | Normal execution | | ||
| | 70-90% of any limit | `[BUDGET WARNING]` | Prioritize remaining work, skip nice-to-haves | | ||
| | 90-100% of any limit | `[BUDGET URGENT]` | Wrap up immediately, write partial artifacts | | ||
| | >100% of any limit | `[BUDGET EXCEEDED]` | Finalize artifact manifest, stop execution | | ||
|
|
||
| ### Context Budget | ||
|
|
||
| When `budget.context-budget` is set, monitor context window usage against the specified fraction. At 70% of the context budget, consider spawning child agents for remaining work rather than consuming more context. | ||
|
|
||
| ### Timeout | ||
|
|
||
| When `budget.timeout-minutes` is set, track wall-clock time from activation. At 80% of timeout, emit `[TIMEOUT WARNING]`. At 100%, finalize and stop. | ||
|
|
||
| ## 3. Required Outputs | ||
|
|
||
| ### Verification Methods | ||
|
|
||
| **existence** (default): Each required output's file path or glob pattern is checked. The output must exist and be non-empty (>0 bytes). | ||
|
|
||
| **content-check**: The LLM reads each output and verifies it matches the `description` field. Score 0.0-1.0 based on completeness and relevance. | ||
|
|
||
| **script**: Run the specified verification script. Exit code 0 = pass, non-zero = fail. | ||
|
|
||
| **llm-judge**: The LLM scores each output for quality against the description. Must meet `verification.threshold`. | ||
|
|
||
| ### Optional Outputs | ||
|
|
||
| Outputs marked `optional: true` are tracked but don't block completion. Missing optional outputs are logged in the manifest with `"exists": false`. | ||
|
|
||
| ### Output Timing | ||
|
|
||
| Required outputs are verified at two points: | ||
| 1. **On-demand**: When the LLM believes the harness is complete | ||
| 2. **On-stop**: When the session ends or budget is exhausted | ||
|
|
||
| ## 4. Permission Composition | ||
|
|
||
| Skill permissions compose with `agent-iam.json` profiles using intersection semantics: | ||
|
|
||
| 1. Start with the IAM profile's permissions (tool access, directory scoping) | ||
| 2. **Add** paths from `permissions.additional-paths` | ||
| 3. **Remove** paths matching `permissions.denied-paths` | ||
| 4. If `can-spawn-agents` is false, deny Agent/Task tool access | ||
| 5. If `can-modify-harness` is false, deny Write/Edit to any `SKILL.md` file | ||
|
|
||
| The narrower permission always wins. A skill cannot grant itself more access than its IAM profile allows. | ||
|
|
||
| ## 5. Child Agent Lifecycle | ||
|
|
||
| ### Spawning | ||
|
|
||
| - Check `budget.max-child-agents` before spawning | ||
| - Each child inherits the parent's permissions (can only narrow, never widen) | ||
| - Children receive a fraction of the parent's remaining budget: `remaining_budget / (max_children - active_children)` | ||
|
|
||
| ### Inheritance | ||
|
|
||
| Children inherit: | ||
| - Permission scope (intersection with parent) | ||
| - Artifact directory (children write to `{parent-artifact-dir}/children/{child-id}/`) | ||
| - Budget fraction (decremented from parent's remaining budget) | ||
|
|
||
| Children do NOT inherit: | ||
| - Parent's tool-call counter (children have their own) | ||
| - Parent's completion conditions (children have task-specific goals) | ||
|
|
||
| ### Completion | ||
|
|
||
| When a child completes: | ||
| 1. Child writes its results as artifacts in its subdirectory | ||
| 2. Parent reads child artifacts via the artifact manifest | ||
| 3. Child's budget usage is added to parent's totals | ||
|
|
||
| ### Failure | ||
|
|
||
| - Child failure does NOT terminate the parent | ||
| - Parent may retry the child (if budget allows) or proceed without the child's output | ||
| - Failed child runs are logged in the parent's execution log | ||
|
|
||
| ## 6. Artifact Lifecycle | ||
|
|
||
| ### Creation | ||
|
|
||
| On skill activation with a contract: | ||
| 1. Create `{state-dir}` directory | ||
| 2. Write `contract-snapshot.json` with the active contract | ||
| 3. Create empty `execution-log.jsonl` | ||
| 4. Create `outputs/` subdirectory | ||
|
|
||
| ### During Execution | ||
|
|
||
| After each significant action, append to `execution-log.jsonl`: | ||
| ```json | ||
| {"timestamp": "...", "action": "tool_call", "tool": "Read", "target": "src/index.ts", "budget_remaining": {"tool_calls": 46}} | ||
| ``` | ||
|
|
||
| ### Sealing | ||
|
|
||
| On completion (or budget exhaustion): | ||
| 1. Verify required outputs | ||
| 2. Compute verification scores | ||
| 3. Write `artifact-manifest.json` with final state | ||
| 4. Write verification results to `verification/` | ||
|
|
||
| ### Retention | ||
|
|
||
| Artifacts older than `artifacts.retention` may be cleaned up on SessionStart. The `_index.json` at `.acos/artifacts/` tracks all runs across skills. | ||
|
|
||
| ## 7. Harness vs. Runtime Boundary | ||
|
|
||
| The charter defines runtime behavior. The skill defines task behavior. | ||
|
|
||
| | Belongs in SKILL.md (Harness) | Belongs in Charter (Runtime) | | ||
| |---|---| | ||
| | What to do (workflow steps) | How contracts are enforced | | ||
| | What outputs to produce | How outputs are verified | | ||
| | What budget limits to set | How budget signals are emitted | | ||
| | What permissions to request | How permissions compose | | ||
| | What agents to spawn | How child lifecycle is managed | | ||
|
|
||
| This boundary enables portable harnesses: the same SKILL.md can execute under different runtimes (Claude Code, Cursor, generic CLI) with different charter implementations. | ||
|
|
||
| ## 8. Backward Compatibility | ||
|
|
||
| - Skills without a `contract:` block are executed exactly as before — no contract parsing, no budget tracking, no artifact management | ||
| - The charter is advisory for the LLM, not programmatically enforcing — Claude reads it and adjusts behavior | ||
| - Existing hooks (`circuit-breaker.sh`, `context-budget-tracker.ts`) continue to operate independently as hard backstops | ||
| - New NLAH hooks (`contract-enforcer.js`, `artifact-manager.js`) are additive to the hook pipeline |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| #!/usr/bin/env node | ||
|
|
||
| // NLAH Artifact Manager — Stop hook | ||
| // Seals artifact manifests when a contracted skill session ends. | ||
| // Reads active contract from .acos/artifacts/_active-contract-{sid}.json, | ||
| // verifies required outputs (with glob support), writes the final manifest, | ||
| // and clears active state. | ||
|
|
||
| const fs = require('fs'); | ||
| const path = require('path'); | ||
|
|
||
| const ACOS_DIR = path.join(process.cwd(), '.acos', 'artifacts'); | ||
| const SESSION_ID = process.env.ACOS_SESSION_ID || process.env.CLAUDE_SESSION_ID || 'default'; | ||
| const ACTIVE_FILE = path.join(ACOS_DIR, `_active-contract-${SESSION_ID}.json`); | ||
| const BUDGET_FILE = path.join(ACOS_DIR, `_budget-state-${SESSION_ID}.json`); | ||
|
|
||
| function globMatch(pattern, cwd) { | ||
| if (!pattern.includes('*') && !pattern.includes('?')) { | ||
| const full = path.join(cwd, pattern); | ||
| if (fs.existsSync(full)) { | ||
| const stat = fs.statSync(full); | ||
| return [{ path: pattern, size: stat.size }]; | ||
| } | ||
| return []; | ||
| } | ||
|
|
||
| const parts = pattern.split('/'); | ||
| let dirs = [cwd]; | ||
| let relPaths = ['']; | ||
|
|
||
| for (const part of parts) { | ||
| const nextDirs = []; | ||
| const nextRels = []; | ||
| for (let i = 0; i < dirs.length; i++) { | ||
| const dir = dirs[i]; | ||
| const rel = relPaths[i]; | ||
| if (!fs.existsSync(dir)) continue; | ||
| let entries; | ||
| try { entries = fs.readdirSync(dir, { withFileTypes: true }); } | ||
| catch { continue; } | ||
| for (const entry of entries) { | ||
| if (matchSegment(part, entry.name)) { | ||
| nextDirs.push(path.join(dir, entry.name)); | ||
| nextRels.push(rel ? `${rel}/${entry.name}` : entry.name); | ||
| } | ||
| } | ||
| } | ||
| dirs = nextDirs; | ||
| relPaths = nextRels; | ||
| } | ||
|
|
||
| return dirs.map((d, i) => { | ||
| try { | ||
| const stat = fs.statSync(d); | ||
| return { path: relPaths[i], size: stat.size }; | ||
| } catch { return null; } | ||
| }).filter(Boolean); | ||
| } | ||
|
|
||
| function matchSegment(pattern, name) { | ||
| if (pattern === '*') return true; | ||
| const re = new RegExp('^' + pattern.replace(/[.+^${}()|[\]\\]/g, '\\$&').replace(/\*/g, '.*').replace(/\?/g, '.') + '$'); | ||
| return re.test(name); | ||
| } | ||
|
|
||
| function main() { | ||
| if (!fs.existsSync(ACTIVE_FILE)) { | ||
| process.exit(0); | ||
| } | ||
|
|
||
| let contract; | ||
| try { | ||
| contract = JSON.parse(fs.readFileSync(ACTIVE_FILE, 'utf8')); | ||
| } catch { | ||
| process.exit(0); | ||
| } | ||
|
|
||
| const stateDir = contract.stateDir; | ||
| if (!stateDir || !fs.existsSync(stateDir)) { | ||
| process.exit(0); | ||
| } | ||
|
|
||
| let budget = { tool_calls: 0, file_edits: 0, child_agents: 0 }; | ||
| try { | ||
| budget = JSON.parse(fs.readFileSync(BUDGET_FILE, 'utf8')); | ||
| } catch {} | ||
|
|
||
| const cwd = process.cwd(); | ||
| const outputs = (contract.contract?.['required-outputs'] || []).map(o => { | ||
| const result = { name: o.name, type: o.type, optional: o.optional || false }; | ||
| if (o.pattern) { | ||
| result.path = o.pattern; | ||
| const matches = globMatch(o.pattern, cwd); | ||
| result.exists = matches.length > 0; | ||
| if (result.exists) { | ||
| result.matched_files = matches.map(m => m.path); | ||
| result.size_bytes = matches.reduce((sum, m) => sum + m.size, 0); | ||
| } | ||
| } else { | ||
| const artifactPath = path.join(stateDir, 'outputs', o.name); | ||
| result.exists = fs.existsSync(artifactPath); | ||
| if (result.exists) { | ||
| try { | ||
| result.size_bytes = fs.statSync(artifactPath).size; | ||
| } catch {} | ||
| } | ||
| } | ||
| return result; | ||
| }); | ||
|
|
||
| const requiredMet = outputs | ||
| .filter(o => !o.optional) | ||
| .every(o => o.exists); | ||
|
|
||
| const budgetLimits = contract.contract?.budget || {}; | ||
| const manifest = { | ||
| skill: contract.skill, | ||
| session_id: contract.sessionId, | ||
| contract_version: '1.0', | ||
| started_at: budget.started_at || null, | ||
| completed_at: new Date().toISOString(), | ||
| status: requiredMet ? 'completed' : 'partial', | ||
| required_outputs: outputs, | ||
| completion_conditions_met: requiredMet, | ||
| budget_usage: { | ||
| tool_calls: { | ||
| used: budget.tool_calls || 0, | ||
| limit: budgetLimits['max-tool-calls'] || null, | ||
| }, | ||
| file_edits: { | ||
| used: budget.file_edits || 0, | ||
| limit: budgetLimits['max-file-edits'] || null, | ||
| }, | ||
| child_agents: { | ||
| used: budget.child_agents || 0, | ||
| limit: budgetLimits['max-child-agents'] || null, | ||
| }, | ||
| }, | ||
| parent_session: null, | ||
| child_sessions: [], | ||
| }; | ||
|
|
||
| const manifestPath = path.join(stateDir, contract.contract?.artifacts?.manifest || 'artifact-manifest.json'); | ||
| fs.mkdirSync(path.dirname(manifestPath), { recursive: true }); | ||
| fs.writeFileSync(manifestPath, JSON.stringify(manifest, null, 2)); | ||
|
|
||
| const indexPath = path.join(ACOS_DIR, '_index.json'); | ||
| let index = []; | ||
| try { | ||
| const data = JSON.parse(fs.readFileSync(indexPath, 'utf8')); | ||
| if (Array.isArray(data)) { | ||
| index = data; | ||
| } | ||
| } catch {} | ||
|
Comment on lines
+148
to
+154
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If let index = [];
try {
const data = JSON.parse(fs.readFileSync(indexPath, 'utf8'));
if (Array.isArray(data)) {
index = data;
}
} catch {} |
||
| index.push({ | ||
| skill: contract.skill, | ||
| session_id: contract.sessionId, | ||
| status: manifest.status, | ||
| completed_at: manifest.completed_at, | ||
| state_dir: stateDir, | ||
| }); | ||
| fs.writeFileSync(indexPath, JSON.stringify(index, null, 2)); | ||
|
|
||
| try { fs.unlinkSync(ACTIVE_FILE); } catch {} | ||
| try { fs.unlinkSync(BUDGET_FILE); } catch {} | ||
|
|
||
| console.log(`[NLAH] Sealed artifact manifest: ${manifest.status} (${outputs.filter(o => o.exists).length}/${outputs.length} outputs)`); | ||
| } | ||
|
|
||
| main(); | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The schema and documentation define
patternas a glob pattern (e.g.,docs/specs/SPEC-*.md). However,fs.existsSyncdoes not support glob expansion and will treat wildcard characters (like*) as literal characters, causing verification to fail for any actual glob patterns. Consider using a glob matching library or implementing a simple directory-scanning helper to resolve wildcard patterns.