Calibration error counts a confidence of exactly 1.0 in an extra bin instead of the last bin

[Nicholas022400701]

🐛 Bug

_binning_bucketize in torchmetrics/functional/classification/calibration_error.py computes the bin index as

indices = torch.bucketize(confidences, bin_boundaries, right=True) - 1

For a confidence of exactly 1.0 (the last boundary) this returns n_bins, one past the last real bin [1 - 1/n_bins, 1]. The three accumulators are allocated with len(bin_boundaries) == n_bins + 1 entries, so instead of an index error the sample is silently counted in a hidden (n_bins + 1)-th bin of its own. Every norm (l1, l2, max) and both the binary and the multiclass metric go through this path.

A confidence of exactly 1.0 is not exotic: in float32 torch.sigmoid(x) == 1.0 for x >= ~17 and softmax saturates the same way, so any reasonably confident model sends part of its predictions into the phantom bin. The result then differs from the textbook definition (Guo et al. 2017, n_bins equal-width bins with the last one closed on the right) and from netcal, which the test-suite uses as reference and which bins with np.histogram semantics.

To Reproduce

import torch
from torchmetrics.functional.classification import binary_calibration_error

preds = torch.tensor([0.95, 1.0])
target = torch.tensor([1, 0])
# both predictions belong to the last of 15 bins: mean confidence 0.975, accuracy 0.5 -> 0.475
print(binary_calibration_error(preds, target, n_bins=15))  # tensor(0.5250)

from netcal.metrics import ECE
print(ECE(15).measure(preds.numpy(), target.numpy()))    # 0.475

Torchmetrics splits the pair into bin 14 (|1 - 0.95| * 0.5 = 0.025) and the phantom bin 15 (|0 - 1| * 0.5 = 0.5) and reports 0.525.

On a larger example (2000 samples, 10 classes, logits N(0, 15) so 225 predictions have max softmax == 1.0) the multiclass MCE is 0.902222 on master versus 0.900395 from netcal; after the fix below it is 0.900396. For l1 the two halves of the last bin often have the same sign of acc - conf, which is why the existing randomised tests never caught this, but as soon as the top bin is slightly under-confident and the 1.0 group is not perfectly accurate the two halves get opposite signs and the reported ECE goes up.

How it got there: #1648 fixed the mirror case (confidence 0.0 was wrapping to index -1) by switching to right=True, which moved the off-by-one to the 1.0 end, and the accumulators were enlarged by one entry so the new index would not raise.

Fix

Bucketize against the inner boundaries only and allocate n_bins accumulators:

num_bins = len(bin_boundaries) - 1
indices = torch.bucketize(confidences, bin_boundaries[1:-1], right=True)

This gives 0 for confidence 0.0, n_bins - 1 for confidence 1.0 and leaves every other input unchanged (same semantics as np.digitize on the inner edges / np.histogram). I have the patch and a test ready locally:

• new test preds=[0.95, 1.0], target=[1, 0], n_bins=15 expecting 0.475 for all three norms: fails on master (0.525), passes with the fix
• tests/unittests/classification/test_calibration_error.py: 390 passed, 16 skipped, 2 xfailed with the fix, including the existing test_binary_with_zero_pred

The output only changes for inputs that contain a confidence of exactly 1.0. I would treat this as a plain bug fix without a flag, but if you prefer to keep the old numbers reproducible behind an option I can do that instead. If you agree with the direction I will open the PR referencing this issue.

Patch

The complete change is below so that nobody has to redo it; I will open it as a PR as soon as a maintainer confirms the direction.

git diff against master (2 files, +18 -5)

diff --git a/src/torchmetrics/functional/classification/calibration_error.py b/src/torchmetrics/functional/classification/calibration_error.py
index 22e52958..04daee81 100644
--- a/src/torchmetrics/functional/classification/calibration_error.py
+++ b/src/torchmetrics/functional/classification/calibration_error.py
@@ -42,11 +42,14 @@ def _binning_bucketize(
 
     """
     accuracies = accuracies.to(dtype=confidences.dtype)
-    acc_bin = torch.zeros(len(bin_boundaries), device=confidences.device, dtype=confidences.dtype)
-    conf_bin = torch.zeros(len(bin_boundaries), device=confidences.device, dtype=confidences.dtype)
-    count_bin = torch.zeros(len(bin_boundaries), device=confidences.device, dtype=confidences.dtype)
-
-    indices = torch.bucketize(confidences, bin_boundaries, right=True) - 1
+    num_bins = len(bin_boundaries) - 1
+    acc_bin = torch.zeros(num_bins, device=confidences.device, dtype=confidences.dtype)
+    conf_bin = torch.zeros(num_bins, device=confidences.device, dtype=confidences.dtype)
+    count_bin = torch.zeros(num_bins, device=confidences.device, dtype=confidences.dtype)
+
+    # bucketize against the inner boundaries only so that a confidence of exactly 1.0 lands in the last bin
+    # instead of in an extra bin that does not exist
+    indices = torch.bucketize(confidences, bin_boundaries[1:-1], right=True)
 
     count_bin.scatter_add_(dim=0, index=indices, src=torch.ones_like(confidences))
 
diff --git a/tests/unittests/classification/test_calibration_error.py b/tests/unittests/classification/test_calibration_error.py
index 9f01adf7..96662c22 100644
--- a/tests/unittests/classification/test_calibration_error.py
+++ b/tests/unittests/classification/test_calibration_error.py
@@ -141,6 +141,16 @@ def test_binary_with_zero_pred():
     assert binary_calibration_error(preds, target, n_bins=2, norm="l1") == torch.tensor(0.6)
 
 
+@pytest.mark.parametrize("norm", ["l1", "l2", "max"])
+def test_binary_with_confidence_one(norm):
+    """Test that a confidence of exactly 1.0 is counted in the last bin together with the other confident preds."""
+    preds = torch.tensor([0.95, 1.0])
+    target = torch.tensor([1, 0])
+    # both predictions fall in the last of 15 bins: mean confidence 0.975, accuracy 0.5, so every norm gives 0.475
+    expected = torch.tensor(0.475)
+    assert torch.allclose(binary_calibration_error(preds, target, n_bins=15, norm=norm), expected)
+
+
 def _reference_netcal_multiclass_calibration_error(preds, target, n_bins, norm, ignore_index):
     preds = preds.numpy()
     target = target.numpy().flatten()

Environment

• TorchMetrics master at 85f4e16 (also present in 1.9.0)
• Python 3.13, PyTorch 2.14.0+cpu, netcal 1.4.0
• Linux

AI disclosure: I used an AI coding agent to help write this patch, the tests and this description. I have read the change and the tests myself and I will answer review comments personally.