Motivation
I maintain ultrafast-pycocotools, a BSD-2-Clause Rust implementation of the COCO evaluation API. I would like to propose it as an explicit, optional third backend for MeanAveragePrecision and the functional mean_average_precision API.
Users would install torchmetrics[ultrafast] and select backend="ultrafast". The default remains pycocotools; backend selection is local to each metric and does not replace process-wide imports. The optional extra requires the published ultrafast-pycocotools >=0.1.11 and torchvision, without requiring either reference evaluator.
Proposed scope
- bbox, segmentation and combined evaluation, including class metrics and extended arrays.
- Existing state lifecycle, JSON conversion and distributed aggregation paths.
- Normalize boolean masks to Fortran-contiguous uint8 before RLE encoding.
- Preserve pycocotools summary semantics, including its fixed maxDets=100 AP behavior with custom detection caps.
I have a concrete implementation ready for a draft PR so the scope can be reviewed. Local MAP tests across all three backends pass (161 passed, 15 CUDA skips), and the 10 existing ultrafast-parameterized distributed tests pass with two CPU/Gloo ranks on Linux. Additional tests compare every returned tensor against pycocotools with zero tolerance across ties, crowds, empty images, incremental updates, reset, pickle and functional evaluation.
I am not claiming a TorchMetrics-level speedup from evaluator-only benchmarks. This proposal is for explicit backend choice; application-level performance can be measured independently.
Motivation
I maintain ultrafast-pycocotools, a BSD-2-Clause Rust implementation of the COCO evaluation API. I would like to propose it as an explicit, optional third backend for
MeanAveragePrecisionand the functionalmean_average_precisionAPI.Users would install
torchmetrics[ultrafast]and selectbackend="ultrafast". The default remainspycocotools; backend selection is local to each metric and does not replace process-wide imports. The optional extra requires the published ultrafast-pycocotools >=0.1.11 and torchvision, without requiring either reference evaluator.Proposed scope
I have a concrete implementation ready for a draft PR so the scope can be reviewed. Local MAP tests across all three backends pass (161 passed, 15 CUDA skips), and the 10 existing ultrafast-parameterized distributed tests pass with two CPU/Gloo ranks on Linux. Additional tests compare every returned tensor against pycocotools with zero tolerance across ties, crowds, empty images, incremental updates, reset, pickle and functional evaluation.
I am not claiming a TorchMetrics-level speedup from evaluator-only benchmarks. This proposal is for explicit backend choice; application-level performance can be measured independently.