RetrievalMRR when non of the targets was correctly retrieved

[matanat]

Hi,
I'm a bit confused about the Mean Reciprocal Rank implementation.
Based on wikipedia:

If none of the proposed results are correct, reciprocal rank is 0

But if I run:

mrr = torchmetrics.RetrievalMRR()

preds = torch.tensor([0.0, 0.0, 0.0])
indexes = torch.tensor([0, 0, 0])
target = torch.tensor([1, 0, 0])

mrr(preds, target, indexes=indexes)

I get tensor(1.) when expecting to get 0.

Am I misunderstanding something?

[celsofranssa]

Hello @matanat,

I had a discussion about it, and I am still confused. Have you tried implementing it by yourself (using these guidelines)?

[Borda]

@matanat — your expected behavior is correct (MRR should be 0 when no relevant docs are retrieved), but the issue in your example is that all predictions have the same score (0.0), which creates an ambiguous ranking.

When all scores are equal, the ranking is arbitrary. TorchMetrics assigns rank 1 to the first element, and since target[0] = 1, it sees the first item as "correctly retrieved at rank 1" → MRR = 1.0.

Fix: use meaningful scores. MRR requires a ranked list:

import torch
from torchmetrics.retrieval import RetrievalMRR

mrr = RetrievalMRR()

# Case 1: relevant item ranked first → MRR = 1.0
preds = torch.tensor([0.9, 0.3, 0.1])
target = torch.tensor([1, 0, 0])
indexes = torch.tensor([0, 0, 0])
print(mrr(preds, target, indexes))  # tensor(1.0)

# Case 2: relevant item ranked last → MRR = 0.333
mrr.reset()
preds = torch.tensor([0.1, 0.5, 0.9])
target = torch.tensor([1, 0, 0])
indexes = torch.tensor([0, 0, 0])
print(mrr(preds, target, indexes))  # tensor(0.3333)

# Case 3: no relevant items → MRR = 0.0
mrr.reset()
preds = torch.tensor([0.9, 0.5, 0.1])
target = torch.tensor([0, 0, 0])
indexes = torch.tensor([0, 0, 0])
print(mrr(preds, target, indexes))  # tensor(0.0)

Summary: MRR = 0 when target is all zeros (no relevant docs). MRR with tied scores is undefined — always provide distinct ranking scores.

Docs: RetrievalMRR