Idea: a devops-bench ↔ EvalPort adapter for portable benchmark results

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange spec (TestCase/Grader/Result/ResultSet/GraderResult, Apache 2.0, Python + TS SDKs) for making LLM/agent eval data portable across frameworks. Posting this as a discussion rather than an issue since it's a "does this interest anyone" question, not a scoped work item yet.

I read the real source before writing this (not just the README): devops_bench/core/results.py's Status enum (PASSED/FAILED/ERROR/SKIPPED) and Result (status, reason, elapsed_sec, details, .ok), plus devops_bench/verification/spec.py's VerificationEntry (name, role: objective|safeguard, severity, weight, check) and a real task (tasks/common/opa-remediation/task.yaml) to see how prompt + verification_spec actually look in practice.

The shapes line up well:

• A task's prompt (rendered) → EvalPort TestCase.input; expected_output (your judge rubric text) → TestCase.expected_output.
• Each VerificationEntry → one EvalPort Grader (type: "custom"), with role/severity/weight carried in grader.params since EvalPort has no first-class equivalent for "safeguard vs. objective, weighted."
• Each entry's per-check Result → one GraderResult: passed = result.ok, score derived from weight when passed, reason = result.reason, with status/elapsed_sec/details preserved under metadata.devops_bench rather than dropped.
• A full run's GraderResult list for a task → EvalPort ResultSet.results[i]; a whole benchmark session → the ResultSet.

The closest real precedent in EvalPort's own adapter set is deepeval-openeval-adapter (verified real code, not a scaffold) — it does the same "many framework-native check objects → one GraderResult each, framework specifics preserved in metadata" translation for DeepEval's MetricData. Your role/severity/weight scoring model is actually closer to what EvalPort's TestCase was shaped for than most adapters need, since nothing here has to be invented.

Rough sketch of the two conversions:

from devops_bench.core.results import Status

def entry_result_to_grader_result(entry, result) -> dict:
    """VerificationEntry + devops-bench Result -> EvalPort GraderResult."""
    return {
        "grader_id": entry.name,
        "type": "custom",
        "score": entry.weight if result.status is Status.PASSED else 0.0,
        "passed": result.ok,
        "reason": result.reason,
        "metadata": {
            "devops_bench": {
                "role": entry.role,
                "severity": entry.severity,
                "weight": entry.weight,
                "status": result.status.value,
                "elapsed_sec": result.elapsed_sec,
                "details": result.details,
            }
        },
    }

def task_to_testcase(task_id, prompt, expected_output, entries) -> dict:
    return {
        "id": task_id,
        "input": prompt,
        "expected_output": expected_output,
        "graders": [e.name for e in entries],
    }

Motivation, concretely: this would let a devops-bench run be reported in a format other eval tooling (or a dashboard, or a comparison against a non-K8s benchmark) can consume without writing devops-bench-specific parsing — and conversely, an EvalPort-shaped suite could seed new devops-bench tasks. Happy to build this as a standalone adapter package (zero footprint on this repo, same pattern as the DeepEval one) and open a PR if there's interest — just didn't want to send a PR cold without checking this is wanted first, especially since I see the CLA requirement in CONTRIBUTING.md.

Does this seem useful, and if so, is a standalone devops-bench-openeval-adapter (living in the EvalPort repo, depending on devops-bench as a library) the right shape, or would maintainers rather see conversion helpers live in-tree here instead?

— Sahi, independent contributor (not affiliated with this project)