Fix COCO export instance segmentation coordinates out of bounds

[qianchongyang]

Problem

When exporting datasets to COCO format, instance segmentation masks have coordinates that exceed the bounds of their bounding boxes. This causes data integrity issues in exported annotations.

Solution

Added clipping logic to ensure segmentation polygon coordinates are clipped to their corresponding bounding box bounds before export.

Validation

# Export COCO dataset
ds.export(export_dir=PATH, dataset_type=fo.types.COCODetectionDataset)

# All segmentation coordinates now within bbox bounds
# No OOB coordinates in exported labels.json

Fixes #2847

Summary by CodeRabbit

• Bug Fixes
  • Improved COCO format exports to correctly handle polyline segmentations by clipping segmentation points to their corresponding bounding box regions.

[coderabbitai]

Walkthrough

The changes fix an out-of-bounds segmentation issue in COCO exports by adding bbox clipping logic to the polyline-to-COCO-segmentation conversion. When COCOObject.from_label converts polyline labels to segmentations, it now passes the computed bounding box to _polyline_to_coco_segmentation, which clips each converted segmentation point to the bbox region before adding it to the polygon coordinates. This prevents segmentation coordinates from extending beyond their corresponding bounding box bounds.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: fixing out-of-bounds segmentation coordinates in COCO export.
Description check	✅ Passed	The description adequately covers the problem, solution, and validation, though it omits the formal PR template sections on testing details and release notes.
Linked Issues check	✅ Passed	The PR directly addresses issue #2847 by implementing coordinate clipping logic to prevent segmentation polygon points from exceeding bounding box bounds during COCO export.
Out of Scope Changes check	✅ Passed	All changes are focused on fixing out-of-bounds segmentation coordinates; no unrelated modifications are present.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

You can get early access to new features in CodeRabbit.

Enable the early_access setting to enable early access features such as new models, tools, and more.

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

fiftyone/utils/coco.py (1)

2253-2277: 🧹 Nitpick | 🔵 Trivial

LGTM — clipping implementation is correct.

The logic properly clips absolute pixel coordinates to the bbox bounds using consistent int() truncation. The optional bbox parameter maintains backwards compatibility, and the iscrowd early-return path correctly skips clipping since it uses RLE representation.

One optional improvement: add a return type annotation per the static analysis hint.

🔧 Optional: Add return type annotation

 def _polyline_to_coco_segmentation(
     polyline, frame_size, iscrowd="iscrowd", bbox=None
-):
+) -> list:

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@fiftyone/utils/coco.py` around lines 2253 - 2277, Add an explicit return type
annotation to _polyline_to_coco_segmentation: change its signature to include a
Union return type reflecting COCO segmentation outputs (e.g.
Union[List[List[int]], Dict[str, Any]]), and add the necessary typing imports
(from typing import List, Dict, Any, Union) at the top of the module; keep the
existing parameters (polyline, frame_size, iscrowd="iscrowd", bbox=None)
unchanged so behavior and compatibility remain the same.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@fiftyone/utils/coco.py`:
- Around line 2253-2277: Add an explicit return type annotation to
_polyline_to_coco_segmentation: change its signature to include a Union return
type reflecting COCO segmentation outputs (e.g. Union[List[List[int]], Dict[str,
Any]]), and add the necessary typing imports (from typing import List, Dict,
Any, Union) at the top of the module; keep the existing parameters (polyline,
frame_size, iscrowd="iscrowd", bbox=None) unchanged so behavior and
compatibility remain the same.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a24aa8d7-c46e-496a-ae15-24bc2d7ba742

📥 Commits

Reviewing files that changed from the base of the PR and between 884f668 and f6373a3.

📒 Files selected for processing (1)

• fiftyone/utils/coco.py

[Burhan-Q]

Thanks for the PR @qianchongyang! There is a proposed fix #6904 from February of this year, and the conversation ended with a pause on merging those changes, to reconcile with voxel51/eta/pull/693. As far as I can tell, there's still some deliberation happening, so you may want to take a look at the other PRs and chime in with your perspective