Skip to content

README.md

Latest commit

 

History

History
133 lines (112 loc) · 4.01 KB

README.md

File metadata and controls

133 lines (112 loc) · 4.01 KB
Raw

Real-time WebRTC VLM Multi-Object Detection Demo

Quick Start

git clone <repo>
cd <repo>
./start.sh
  • Open http://localhost:3000 on your laptop.
  • Scan the QR code with your phone to join and stream video.
  • Allow camera access on your phone.
  • You should see your phone video mirrored on the laptop with overlays.

Phone-Join Instructions

  • After starting the project, use the PUBLIC_URL displayed in the terminal.
  • To generate a QR code for easy phone access, run: 
    echo $PUBLIC_URL | qrencode -o qr.png
  • Open the qr.png file and scan it with your phone to join.

Modes

  • WASM mode (default): All inference runs in-browser (no server GPU required).
  • Server mode: Inference runs on backend (CPU-only, no GPU required).
  • Switch mode: MODE=wasm ./start.sh or MODE=server ./start.sh

Metrics Generation

  • To collect metrics, run the benchmarking script: 
    ./bench/run_bench.sh --duration 30 --mode wasm
  • The script will generate a metrics.json file with the following details:
    • Median and P95 end-to-end latency
    • Processed FPS
    • Uplink and downlink bandwidth (kbps)

Troubleshooting

  • If phone won’t connect: ensure phone and laptop are on same network OR use ./start.sh --ngrok and use the public URL.
  • If overlays are misaligned: check timestamps in JSON messages.
  • If CPU is high: reduce resolution or use WASM mode.

Project Submission: Real-time WebRTC VLM Multi-Object Detection

1. Git Repository

The project repository includes:

  • Frontend: A Vite-based application for phone-to-browser video streaming.
  • Backend (Optional): A FastAPI server for inference in server mode.
  • Docker Support:
    • Dockerfile for both frontend and backend.
    • docker-compose.yml for local development and deployment.
  • Convenience Script: start.sh to simplify starting the project with options for ngrok tunneling and mode switching.

2. README.md

The README.md provides:

  • One-Command Start Instructions: 
    ./start.sh
  • Mode Switching:
    • WASM mode (default): MODE=wasm ./start.sh
    • Server mode: MODE=server ./start.sh
  • Phone-Join Instructions:
    • Use the PUBLIC_URL displayed in the terminal.
    • Generate a QR code for easy access: 
      echo $PUBLIC_URL | qrencode -o qr.png
    • Scan the QR code with your phone to join.

3. Metrics

The metrics.json file is generated by running the benchmarking script:

./bench/run_bench.sh --duration 30 --mode wasm

Example metrics:

{
  "median_latency_ms": 120,
  "p95_latency_ms": 180,
  "fps": 15.2,
  "uplink_kbps": 350,
  "downlink_kbps": 400
}

4. Loom Video

A 1-minute Loom video demonstrates:

  • Phone-to-browser live overlay.
  • Metrics output.
  • Suggested improvement: Adaptive FPS based on latency and CPU load.

5. Report

The report.md explains:

  • Design Choices:
    • WebRTC for real-time streaming.
    • WASM mode for in-browser inference.
    • Server mode for backend inference.
  • Low-Resource Mode:
    • Optimized for modest hardware (e.g., i5, 8GB RAM).
    • Downscaled input (320x240) and quantized models.
  • Backpressure Policy:
    • Fixed-length frame queue, dropping oldest frames when overloaded.
    • Processes only the latest frame for inference.

Laptop Connectivity Problem

Issue:

  • Problem: Phones may fail to connect to the laptop due to network restrictions or lack of HTTPS.
  • Symptoms: WebRTC signaling fails, or the phone cannot establish a connection.

Solutions:

  1. Use ngrok:

    • Start the project with ngrok tunneling: 
      ./start.sh --ngrok
    • Use the public HTTPS URL provided by ngrok.

  2. Ensure Same Network:

    • Verify that the phone and laptop are on the same Wi-Fi network.

  3. Browser Flags:

    • For Chrome, use the --unsafely-treat-insecure-origin-as-secure flag: 
      chrome --unsafely-treat-insecure-origin-as-secure=http://<laptop-ip>:3000

  4. Alternative Browsers:

    • Use Firefox, which is more permissive with local IPs.