Add skill evaluation harness (evals) with per-skill test cases - #15
Draft
claude[bot] wants to merge 2 commits into
Draft
Add skill evaluation harness (evals) with per-skill test cases#15claude[bot] wants to merge 2 commits into
claude[bot] wants to merge 2 commits into
Conversation
Adds a with/without-skill evaluation harness so we can measure whether a skill actually improves the model's answers, implementing the agentskills.io "evaluating skills" methodology (issue #11). Methodology: each test case is run twice against the same prompt -- once WITH the skill injected as context (SKILL.md + references/) and once WITHOUT it (a neutral baseline). An LLM judge grades every assertion PASS/FAIL with evidence (no benefit of the doubt), and results are aggregated per iteration into a with-vs-without pass-rate delta to iterate on. CLI (invoked as a module, no console script): uv run python -m evals.run_evals --skill <name> uv run python -m evals.run_evals --all [--no-baseline] [--fail-under 0.6] The evals/ package: schema (dependency-free validator + JSON Schema), client (lazy Anthropic client + retry/backoff), grader (LLM judge), aggregate (artifact writers + benchmark), run_evals (entrypoint). CI: new .github/workflows/evals.yml (separate from validate.yml) runs on workflow_dispatch and on pull_request touching skills/** or evals/**. It evaluates only the changed skills (all skills when the harness itself changes), appends a delta table to the job summary, and uploads the run workspace. Report-only by default -- won't hard-fail builds unless the --fail-under gate is opted in. Test cases: 12 skills x 3 cases each, with checkable assertions grounded in each skill's SKILL.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015UTmT9HV4i7TkM5wVSTGQX
…el run on secret The `evals` PR check failed at the "Run evals" step: run_evals.main() calls build_client(), which raises MissingAPIKeyError and returns exit code 2 when ANTHROPIC_API_KEY is unset. The repo has no such secret, so every PR failed. (`uv sync`/skills-ref installed fine — that step succeeded.) Split evals.yml into two tiers: - Tier 1 (always): a key-free step validates every skills/*/evals/evals.json with the dependency-free evals/schema.py validator on system Python (no uv sync, no SDK, no secret). Hard-fails on any malformed file. - Tier 2 (conditional): a "Detect API key" step exposes has_key via `secrets.ANTHROPIC_API_KEY != ''`; the uv setup, dependency install, and the with/without model eval run are all gated on has_key == 'true'. When the key is absent the job prints "ANTHROPIC_API_KEY not configured — skipping model eval run, ran schema validation only" and stays green. The model run remains report-only (the --fail-under gate stays commented/opt-in). Update evals/README.md CI section to document the two-tier behaviour. validate.yml is untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Ante Javor · Slack thread
Implements the agentskills.io "evaluating skills" methodology so we can measure whether a skill actually improves the model's answers. Closes/relates to #11.
What changes
Before: we had no way to tell whether a skill genuinely improves the model's answers —
validate.ymlonly checks that a skill's structure is well-formed.After:
uv run python -m evals.run_evals --skill <name>runs each of that skill's test cases twice — once with the skill injected as context and once without it (a neutral baseline) — an LLM judge grades each assertion, and you get a with-vs-without pass-rate delta to iterate on. A newevals.ymlCI job runs the same thing, triggerable from the Actions tab (workflow_dispatch) or automatically on PRs that touch skills.How it works
evals/packageschema.py— dependency-freeevals.jsonvalidator + typed dataclasses (fails fast with clear messages before any API call). Mirrored byevals.schema.json(draft-07, for editor tooling) andevals.template.json(copy-paste starter).client.py— Anthropic client construction (clear error whenANTHROPIC_API_KEYis missing) + retry/backoff on transient errors. The SDK is imported lazily, so--helpand validation work with no key installed.grader.py— the LLM judge: one structured call per assertion →PASS/FAIL+ evidence, with a strict "no benefit of the doubt" prompt.aggregate.py— writes per-run artifacts and builds the iteration-levelbenchmark.json(with/without means, stddev, deltas) plus a non-clobberingfeedback.jsontemplate.run_evals.py— CLI entrypoint that ties it together and prints the with-vs-without summary table.skills/<skill>/evals/evals.json.skills/<skill>-workspace/iteration-N/<eval-case>/{with_skill,without_skill}/withoutputs/,timing.json,grading.json, and iteration-levelbenchmark.json+feedback.json.--fail-undergate is report-only by default — the harness always exits 0 unless you explicitly pass a threshold.CI / secrets
evals.ymlneeds anANTHROPIC_API_KEYrepo secret to run.--fail-under 0.6gate line is included to opt in later.workflow_dispatchyou pick a skill orall.Test cases
12 skills × 3 cases each, with checkable assertions grounded in each skill's
SKILL.md:memgraph-brand-ui,memgraph-cpp-query-modules,memgraph-cypher-syntax,memgraph-database-configuration,memgraph-graph-rag,memgraph-indexes-and-constraints,memgraph-lab-write-gss,memgraph-mgconsole-cli,memgraph-model-graph-data,memgraph-python-query-modules,memgraph-run-mage-algorithms,memgraph-rust-query-modules.🤖 Generated with Claude Code
https://claude.ai/code/session_015UTmT9HV4i7TkM5wVSTGQX
Generated by Claude Code