EvalPort adapter for ClawBench's run-meta.json / rescore-summary.json results

[adhabnr-ux]

Hi — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open interchange spec + SDK (evalport-sdk on PyPI, openeval.validate.validate_suite() / validate_result_set()) for making per-example eval results portable across benchmarks and tools — one shared JSON shape a result can round-trip through instead of every consumer writing its own converter.

I read through docs/scoring.md and the runner (src/clawbench/runner/run_support/metadata.py, src/clawbench/runner/judge.py). ClawBench already produces a clean per-task record once a run finishes. make_run_meta() writes run-meta.json with instruction, model, harness, intercepted, result_category, failure_category, adjusted_eligible, and — per docs/scoring.md — the two-stage scoring result gets merged in as judge_match and final_pass (final_pass = intercepted AND judge_match). A batch rolls these up into rescore-summary.json with n_total, n_intercepted, n_judge_match, pass_rate_with_judge, and a tasks: [...] list of per-task records.

That per-task record (instruction as input, final_pass as pass/fail, intercepted/judge_match/result_category as metadata) maps directly onto an EvalPort result set. And since ClawBench already ships import-side adapters the other direction (clawbench-harbor-adapt, clawbench-edgebench-adapt in pyproject.toml's [project.scripts]), an EvalPort adapter would be a natural export-side complement for anyone comparing a ClawBench run against results from another agent benchmark.

Sketch, reading a rescore-summary.json:

# clawbench-openeval-adapter/src/clawbench_openeval_adapter/__init__.py
import json
from pathlib import Path
from openeval.validate import validate_result_set

def to_openeval(rescore_summary_path: str, suite_id: str) -> dict:
    summary = json.loads(Path(rescore_summary_path).read_text())
    examples = []
    for task in summary["tasks"]:
        examples.append({
            "id": task.get("test_case"),
            "input": task.get("instruction"),
            "passed": task.get("final_pass"),
            "metadata": {
                "intercepted": task.get("intercepted"),
                "judge_match": task.get("judge_match"),
                "result_category": task.get("result_category"),
                "model": task.get("model"),
                "harness": task.get("harness"),
            },
        })
    result_set = {"suite_id": suite_id, "examples": examples}
    validate_result_set(result_set)
    return result_set

I pulled test_case/instruction/model/harness/intercepted/result_category directly from make_run_meta() and judge_match/final_pass from the JSON shown in docs/scoring.md, but I haven't run clawbench-rescore myself, so I'd confirm the exact per-task keys inside rescore-summary.json["tasks"] against a real summary file before opening a PR — happy to also pull judge.reason from data/judge.json per-run as a rationale field if that's preferred over whatever's flattened into the summary.

I'd build this as adapters/clawbench-openeval-adapter/ (standalone package, pyproject.toml depending on evalport-sdk + the pinned self-referencing extra convention from Discussion #13, tests, README — zero footprint on clawbench-eval itself) and submit it as a PR. Let me know if useful, or if there's a preferred internal export point instead.

— Sahi (independent contributor, not affiliated with TIGER-AI-Lab)

[reacher-z]

Thank you! We will review this issue soon.

[adhabnr-ux]

Thanks — no rush, and happy to answer any questions on the EvalPort side (spec, mapping, adapter shape) if anything comes up while you're looking at it.

— Sahi

[Perry2004]

Hi Sahi, thanks for the suggestion.

May I get some clarifications on what the "adaptor" you're mentioning is for?
The current adaptors are used to convert the test case definitions, but looks like the EvalPort adaptor you're proposing is for converting the test results?

[adhabnr-ux]

Good question, and yes — you've got it exactly right: this is the results side, not the test-case side.

The distinction, mapped to what's already in the repo:

• clawbench-harbor-adapt / clawbench-edgebench-adapt convert someone else's test case definitions (Harbor's, EdgeBench's) into ClawBench's own eval format, so ClawBench can run them. That's the "import a suite" direction — EvalPort calls this document type a TestCase/EvalSuite.
• What I'm proposing is the reverse, and it's about results, not inputs: take the run-meta.json / rescore-summary.json that a ClawBench run already produces — instruction, final_pass, intercepted, judge_match, result_category, per docs/scoring.md — and map that finished output into EvalPort's ResultSet shape. EvalPort calls this document type ResultSet: one JSON object per benchmark run, with results: [{test_case_id, passed, grader_results, ...}] plus summary stats (I'd flesh out the sketch in the opening post to match this full shape — my first pass simplified the field names for illustration).

Why that's useful: a ResultSet is just a common shape so a run's output — pass/fail, scores, metadata — can be read the same way regardless of which benchmark produced it. If someone wants to put a ClawBench run next to a run from another agent benchmark (Harbor, EdgeBench, whatever else speaks EvalPort) in one dashboard or comparison table, they'd otherwise have to hand-write a converter for ClawBench's specific JSON shape. This adapter does that conversion once, so ClawBench doesn't need to.

What would actually change for ClawBench: nothing, internally. This is additive/optional, not a replacement for anything:

• run-meta.json and rescore-summary.json keep their exact current shape — the runner, make_run_meta(), judge.py, the two-stage scoring pipeline, all untouched.
• The adapter is a separate, standalone package (adapters/clawbench-openeval-adapter/) that only reads those files after a run finishes and writes a new, separate resultset.json alongside them if and when someone chooses to run it.
• Nothing in a normal clawbench-eval / clawbench-rescore invocation changes, imports the adapter, or depends on it. Same relationship the harbor/edgebench adapters already have with clawbench-eval — just running the other direction.

If that answers it, I'm still happy to open the PR — I'd want to run clawbench-rescore on a real fixture first to confirm the exact keys under tasks[] before finalizing the field mapping, rather than going purely off docs/scoring.md. Let me know if that's the right shape to build against, or if there's a preferred internal hook instead.

— Sahi

[Perry2004]

Then that would be fine.
Feel free to open a PR regarding this to match your desired shape.

Since it's not an actual benchmark adaptor but a simple schema conversion, I'd suggest a simple script inside script/ to convert clawbench's result Json schema to EvalPort's.