Real-time WebRTC VLM Multi-Object Detection Demo

Quick Start

git clone <repo>
cd <repo>
./start.sh

• Open http://localhost:3000 on your laptop.
• Scan the QR code with your phone to join and stream video.
• Allow camera access on your phone.
• You should see your phone video mirrored on the laptop with overlays.

Phone-Join Instructions

• After starting the project, use the PUBLIC_URL displayed in the terminal.
• To generate a QR code for easy phone access, run:

  echo $PUBLIC_URL | qrencode -o qr.png

• Open the qr.png file and scan it with your phone to join.

Modes

• WASM mode (default): All inference runs in-browser (no server GPU required).
• Server mode: Inference runs on backend (CPU-only, no GPU required).
• Switch mode: MODE=wasm ./start.sh or MODE=server ./start.sh

Metrics Generation

• To collect metrics, run the benchmarking script:

  ./bench/run_bench.sh --duration 30 --mode wasm

• The script will generate a metrics.json file with the following details:
  • Median and P95 end-to-end latency
  • Processed FPS
  • Uplink and downlink bandwidth (kbps)

Troubleshooting

• If phone won’t connect: ensure phone and laptop are on same network OR use ./start.sh --ngrok and use the public URL.
• If overlays are misaligned: check timestamps in JSON messages.
• If CPU is high: reduce resolution or use WASM mode.

Project Submission: Real-time WebRTC VLM Multi-Object Detection

1. Git Repository

The project repository includes:

• Frontend: A Vite-based application for phone-to-browser video streaming.
• Backend (Optional): A FastAPI server for inference in server mode.
• Docker Support:
  • Dockerfile for both frontend and backend.
  • docker-compose.yml for local development and deployment.
• Convenience Script: start.sh to simplify starting the project with options for ngrok tunneling and mode switching.

2. README.md

The README.md provides:

• One-Command Start Instructions:

  ./start.sh

• Mode Switching:
  • WASM mode (default): MODE=wasm ./start.sh
  • Server mode: MODE=server ./start.sh
• Phone-Join Instructions:
  • Use the PUBLIC_URL displayed in the terminal.
  • Generate a QR code for easy access:

    echo $PUBLIC_URL | qrencode -o qr.png

  • Scan the QR code with your phone to join.

3. Metrics

The metrics.json file is generated by running the benchmarking script:

./bench/run_bench.sh --duration 30 --mode wasm

Example metrics:

{
  "median_latency_ms": 120,
  "p95_latency_ms": 180,
  "fps": 15.2,
  "uplink_kbps": 350,
  "downlink_kbps": 400
}

4. Loom Video

A 1-minute Loom video demonstrates:

• Phone-to-browser live overlay.
• Metrics output.
• Suggested improvement: Adaptive FPS based on latency and CPU load.

5. Report

The report.md explains:

• Design Choices:
  • WebRTC for real-time streaming.
  • WASM mode for in-browser inference.
  • Server mode for backend inference.
• Low-Resource Mode:
  • Optimized for modest hardware (e.g., i5, 8GB RAM).
  • Downscaled input (320x240) and quantized models.
• Backpressure Policy:
  • Fixed-length frame queue, dropping oldest frames when overloaded.
  • Processes only the latest frame for inference.

Laptop Connectivity Problem

Issue:

• Problem: Phones may fail to connect to the laptop due to network restrictions or lack of HTTPS.
• Symptoms: WebRTC signaling fails, or the phone cannot establish a connection.

Solutions:

1. Use ngrok:

   • Start the project with ngrok tunneling:

     ./start.sh --ngrok

   • Use the public HTTPS URL provided by ngrok.

2. Ensure Same Network:

   • Verify that the phone and laptop are on the same Wi-Fi network.

3. Browser Flags:

   • For Chrome, use the --unsafely-treat-insecure-origin-as-secure flag:

     chrome --unsafely-treat-insecure-origin-as-secure=http://<laptop-ip>:3000

4. Alternative Browsers:

   • Use Firefox, which is more permissive with local IPs.