Mirra

Reality-to-Simulation via 3D Semantic Reconstruction

A state-of-the-art pipeline extracting real-world video into metric-semantic 3D environments using DUSt3R geometry and SAM 2 segmentation.

Visual Results

Quick Start

1. Clone

git clone https://github.com/your-org/mirra.git
cd mirra

2. Backend (FastAPI ML Pipeline)

Powered by uv for seamless dependency resolution.

# Sync environment and start the REST API
pip install uv
uv sync

# For AMD ROCm/HIP acceleration (ROCm 6.1):
# uv pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.1

# Run the REST API
uv run uvicorn src.api.server:app --host 0.0.0.0 --port 8000 --reload

3. Frontend (React/Three.js Viewer)

Powered by bun for blazingly fast execution.

cd frontend

bun install
bun run start

# Or you can always use
npm install
npm start

Repository Structure

├── data/                    # Raw inputs and processed staging
│   ├── processed/
│   └── raw/
├── frontend/                # React / Three.js structural viewer
│   ├── public/
│   └── src/
├── outputs/                 # Final generated semantic clouds (.ply)
│   └── final/
└── src/                     # Core ML reconstruction pipeline
    ├── api/                 # FastAPI REST framework
    ├── fusion_engine.py     # 3D projection & label voting
    ├── geometry_engine.py   # DUSt3R depth and pose estimation
    └── semantic_engine.py   # SAM 2 video segmentation

Third-Party Dependencies

Mirra uses the following open-source projects:

• DUSt3R — Licensed under
• Segment Anything (Meta)

These projects remain under their respective licenses. Mirra does not claim ownership of these components. Mirra is licensed under MIT.

Mirra depends on DUSt3R (CC BY-NC-SA 4.0), which restricts commercial usage.

Users are responsible for complying with DUSt3R's license.