Proposal: torchmetrics-openeval-adapter (text metrics ↔ EvalPort result format)

[adhabnr-ux]

EvalPort (https://github.com/adhabnr-ux/evalport) is an open interchange spec + SDK for portable LLM/eval datasets and results — a JSON schema for test suites and result sets, validated through openeval.validate.validate_suite() / validate_result_set() in the evalport-sdk package on PyPI. The idea is that eval data (test cases, graders, scored results) can move between frameworks instead of being locked into one tool's format.

I think torchmetrics' text metrics are a genuinely clean fit, and wanted to float the idea here before building anything, per the "submit an issue, discuss scope" guidance in CONTRIBUTING.md — though to be clear, this wouldn't touch torchmetrics' own code at all. It'd be a standalone adapter package living in the EvalPort repo, zero footprint on this repo.

Why it fits

Metric (src/torchmetrics/metric.py) is torchmetrics' base class, and every text metric under src/torchmetrics/text/ — BLEUScore, ROUGEScore, CHRFScore, CharErrorRate, Perplexity, BERTScore, etc. — follows the same update(preds, target) / compute() -> Tensor shape. That single-tensor-per-call return is exactly what maps onto EvalPort's GraderResult.score (a float in [0, 1]), and a handful of the most common text metrics — BLEU, ROUGE variants, chrF/100 — already report natively in that range, so no lossy rescaling is needed for those.

Before / after sketch, using BLEUScore's own real usage (from bleu.py's docstring):

from torchmetrics.text import BLEUScore

bleu = BLEUScore()
preds = ["the cat is on the mat"]
target = [["there is a cat on the mat", "a cat is on the mat"]]
score = bleu(preds, target)  # tensor(0.7598)

With the proposed adapter:

from torchmetrics_openeval_adapter import metric_result_to_openeval

result_set = metric_result_to_openeval(
    metric_name="bleu",
    predictions=preds,
    references=target,
    item_scores=[float(score)],
    suite_id="translation_suite",
    run_id="run_1",
    started_at="2026-08-23T00:00:00Z",
)
# result_set is a plain dict conforming to EvalPort's ResultSet schema —
# openeval.validate.validate_result_set(result_set) passes.

One honesty note I'd document rather than paper over: torchmetrics' update/compute accumulates n-gram counts across calls (see BLEUScore's numerator/denominator state), so calling compute() on a single example gives a real BLEU number, but it isn't identical in meaning to a true corpus-level BLEU computed over the whole batch — the same caveat that applies to any sentence-level BLEU. The adapter would report both: the real per-example score for GraderResult.score, and the real whole-batch compute() result preserved in metadata, so nothing is fabricated or interpolated. Metrics without a natural [0, 1] range (Perplexity, WER, CER) would map to EvalPort's custom grader type with the raw value carried in metadata rather than being force-clamped into a misleading score.

If this sounds useful, I'd build torchmetrics-openeval-adapter (mirroring the existing huggingface-evaluate-openeval-adapter in structure — pyproject.toml depending on evalport-sdk plus a self-referencing pinned torchmetrics extra, to_openeval()/from_openeval(), tests, README) and submit it as a PR to the EvalPort repo, not to torchmetrics. No action needed on your end beyond a thumbs-up (or a correction if I've got the Metric shape wrong somewhere).

— Sahi, independent contributor, maintaining EvalPort