Evals

A standalone TypeScript eval harness for LLM orchestrator workflows. Two ways to use it:

1. Run locally as a CLI — from this repo, test the built-in components against tasks.yaml
2. Import as a package — from any app, define your own components and run evals programmatically

---

Quick start (CLI — run from this repo)

# 1. Install dependencies
npm install

# 2. Set your API key
cp .env.example .env
# edit .env and add: ANTHROPIC_API_KEY=sk-ant-...

# 3. Run the eval suite
npx tsx main.ts                           # all 8 tasks, 3 runs each, $1 budget
npx tsx main.ts --runs 1 --budget 0.25   # faster smoke check
npx tsx main.ts --component planner      # planner tasks only
npx tsx main.ts --report                 # re-render last result, no API calls
npx tsx main.ts --report results/run-X.jsonl  # specific file

CLI flags

Flag	Default	Description
--suite <path>	tasks.yaml	YAML task file to run
--runs <n>	3	Runs per task (more = tighter CI)
--component <name|all>	all	Filter to one component
--budget <usd>	1.0	Hard cost cap — throws if exceeded
--report [file]	—	Render report only, no new LLM calls

---

Using from another app (programmatic API)

1. Install the package

# local development (evals repo is a sibling directory)
npm install file:../evals

2. Set your API key

Create a .env file in your app:

ANTHROPIC_API_KEY=sk-ant-...

3. Define a component and run evals

import "dotenv/config";
import { z } from "zod";
import { runEval, renderReport, LlmClient } from "@yourorg/evals";
import type { Component, ComponentResult, RunContext } from "@yourorg/evals";

const myComponent: Component<{ query: string }, { answer: string }> = {
  name: "myComponent",
  inputSchema: z.object({ query: z.string() }),
  outputSchema: z.object({ answer: z.string() }),

  async run(input, ctx): Promise<ComponentResult<{ answer: string }>> {
    const llm = new LlmClient(ctx);
    const answer = await llm.complete([
      { role: "user", content: input.query },
    ]);
    return {
      output: { answer },
      trajectory: {
        componentName: "myComponent",
        startedAt: new Date().toISOString(),
        endedAt: new Date().toISOString(),
        llmCalls: [], toolCalls: [], children: [],
        costUsd: 0, tokensIn: 0, tokensOut: 0,
      },
    };
  },
};

const summary = await runEval([myComponent], {
  suite: "./tasks.yaml",   // or pass a Task[] array directly
  runs: 3,
  budgetUsd: 0.50,
});

await renderReport(summary.outputPath);

4. Run the eval script

npx tsx eval.ts
# or, if you added it to package.json scripts:
npm run eval

See the example/ directory for a working Next.js app that uses this pattern.

---

Rubric kinds

Kind	When to use
exact_match	Output must equal a specific value exactly
schema_valid	Output must parse against a zod schema
checklist	LLM judge confirms each listed fact is present
llm_judge	Freeform criteria, scored 0–1 by an LLM judge

---

Adding a component

1. Implement Component<I, O> with a name, inputSchema, outputSchema, and run()
2. Register it in main.ts (for CLI use) or pass it directly to runEval()
3. Add tasks targeting your component name to tasks.yaml or a Task[] array

The runner validates all I/O against your zod schemas and throws loudly on mismatches.

---

Output

Results are streamed to results/run-<timestamp>.jsonl — one JSON object per run. The report reads any JSONL file and renders pass rates with 95% bootstrap confidence intervals.