Mean Average Precision producing incorrect values, maybe?

[dominic-simon]

Hello,

I'm trying to find the Mean Average Precision (mAP) over a set of images from the MSCOCO dataset that have been modified to cause the object detector in use (Faster-RCNN, in this case) to predict objects that don't exist. To be clear, the objects that don't exist are objects that generally have very little to no overlap with any ground truth objects.

My issue is that when I compare the original image mAP to the modified image mAP, they are the exact same, which does not seem correct. Below are what the images look like:
Ground Truths

Original Image Predictions

Modified Image Predictions

To the best of my knowledge, these two images should have two different mAPs, with the original image having a much higher mAP than the modified image since the modified image has many False Positive predictions (all the boxes to the left of the man on the surfboard). However, that is not the case, with both images receiving a mAP of 0.85.

Can someone confirm that I am correct in thinking that these two images should receive different mAPs? If I'm wrong, then perhaps an explanation of where I'm going wrong would be useful? I'm not sure what I'm misunderstanding here, so any help is appreciated.

Here's my code, in case I'm doing something wrong there:

import torch
import torchvision.transforms as transforms
import torchvision.models.detection as models
from torchvision.datasets import CocoDetection
from torchmetrics.detection.mean_ap import MeanAveragePrecision

import os
import numpy as np
from PIL import Image
from tqdm import tqdm
import matplotlib.pyplot as plt

PATCH_SIZE = 100
FOLDER_PATH = f'./SAC_PGD/{PATCH_SIZE}'
DEVICE = 'cuda'

model = models.fasterrcnn_resnet50_fpn_v2(weights='FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1').eval().to(DEVICE)

mAP = MeanAveragePrecision()

ms_coco = CocoDetection('../datasets/MSCOCO/val2017', '../datasets/MSCOCO/annotations/instances_val2017.json')

for i in tqdm(range(14, 15)):
    # Reformatting the mscoco ground truths
    b = []
    l = []
    s = []
    for j in range(len(ms_coco[i][1])):
        bboxes = ms_coco[i][1][j]['bbox']
        corrected = [bboxes[0], bboxes[1], bboxes[0]+bboxes[2], bboxes[1]+bboxes[3]]
        b.append(corrected)
        l.append(ms_coco[i][1][j]['category_id'])
        s.append(1.0)
    gt = [{'boxes':torch.from_numpy(np.array(b)).to(DEVICE),
           'labels':torch.from_numpy(np.array(l)).to(DEVICE),
           'scores':torch.from_numpy(np.array(s)).to(DEVICE)}]

   # Open the original image 
    original = (np.asarray(ms_coco[i][0]) / 255).astype(np.float32)
    row, col = original.shape[0], original.shape[1]

    to_tensor = transforms.Compose([transforms.Resize((row, col)),
                                    transforms.ToTensor()])

   # Open the modified image and a mask showing where the modification is
    image_path = os.path.join(FOLDER_PATH, f'{i}_adv.png')
    mask_path = os.path.join(FOLDER_PATH, f'{i}_mask.png')
    image = Image.open(image_path).convert('RGB')
    mask = Image.open(mask_path).convert('L')
    image = np.transpose(to_tensor(image).numpy(), (1, 2, 0))
    mask = np.stack([to_tensor(mask).numpy()] * 3, axis=3)[0]

    # I was having some issues with data. This solves that 
    image = np.where(mask == 1, original, image)

   # Reformat modified image for use in pytorch Faster-RCNN
    image = torch.from_numpy(np.transpose(image, (2, 0 ,1))).to(DEVICE)

   # Get modified and original image predictions
    with torch.no_grad():
        output = model([image])
        b_out = model([torch.from_numpy(np.transpose(original, (2, 0 ,1))).to(DEVICE)])

   # Draw the bounding boxes onto the images
    from PIL import ImageDraw
    to_pil = transforms.ToPILImage()
    original = to_pil((original * 255).astype(np.uint8))
    attacked_image = to_pil((image * 255).type(torch.uint8))
    drawer = ImageDraw.Draw(original)
    drawer_copy = ImageDraw.Draw(attacked_image)
    bboxes = output[0]['boxes']
    bbboxes = b_out[0]['boxes']
    for box in bboxes:
        drawer_copy.rectangle([box[0], box[1], box[2], box[3]], outline=(255, 0, 0), width=3)
    #for box in ms_coco[i][1]:
    for box in bbboxes:
        #box = box['bbox']
        drawer.rectangle([box[0], box[1], box[2], box[3]], outline=(0, 255, 0), width=3)
    original.save(f'./{i}_benign.png')
    attacked_image.save(f'./{i}_adv.png')

   # Update the predictions used in the mAP. 
   # I switch b_out and output to test the original and modified images mAP, respectively. 
    mAP.update(b_out, gt)

# Compute and show the final mAP.
total_mAP = mAP.compute()
print(total_mAP['map'])

[Leonnorblad]

Any updates on this issue? I have a similar problem

[Borda]

@dominic-simon @Leonnorblad — the MAP values being identical for original vs. adversarial images likely comes down to confidence score filtering, not a bug in the metric.

Here's what's probably happening: Faster R-CNN outputs predictions sorted by confidence. When you have both true detections (high confidence) and adversarial false positives (which may also have high confidence or — more commonly — lower confidence), MAP is dominated by the high-confidence true detections because:

1. AP is precision-recall area — if the true objects are detected at high confidence, they appear first in the ranked list and precision stays high
2. Low-confidence false positives only lower precision at higher recall thresholds, which has a small effect on the area

Debugging steps:

from torchmetrics.detection import MeanAveragePrecision

metric = MeanAveragePrecision(
    iou_type="bbox",
    class_metrics=True,
    extended_summary=True,
)
metric.update(preds_original, targets)
result_orig = metric.compute()
metric.reset()

metric.update(preds_adversarial, targets)
result_adv = metric.compute()

# Compare per-class
print("Original per-class:", result_orig["map_per_class"])
print("Adversarial per-class:", result_adv["map_per_class"])

# Check if the predictions are actually different
print("Num preds original:", sum(len(p["scores"]) for p in preds_original))
print("Num preds adversarial:", sum(len(p["scores"]) for p in preds_adversarial))

Also check your max_detection_thresholds — by default it's [1, 10, 100]. If both original and adversarial produce >100 detections, only the top 100 by confidence are used. If the top 100 are identical, MAP will be the same.

Filter by confidence to see the effect:

threshold = 0.3
filtered_preds = []
for p in preds_adversarial:
    mask = p["scores"] > threshold
    filtered_preds.append({
        "boxes": p["boxes"][mask],
        "scores": p["scores"][mask],
        "labels": p["labels"][mask],
    })

Docs: MeanAveragePrecision