ShotTracker

Computer vision that watches basketball videos and classifies every shot as a make or miss.

---

Results

Metric	Value
YOLO mAP50	0.846
Training images	27,200
Detection classes	ball, human, rim, made, shoot

Optimized for side-angle tripod footage. Accuracy numbers will be updated after retesting with the latest classifier tuning.

---

How It Works

flowchart LR
    A[Mobile App] --> B[Cloudflare R2\nVideo Storage]
    A --> C[FastAPI\nDigital Ocean]
    C --> D[RunPod\nServerless GPU]
    D --> B
    D --> E[ML Worker]
    E --> F[Supabase\nResults + Auth]
    F --> A

    subgraph E[ML Worker]
        direction TB
        E1[FFmpeg Frame\nExtraction] --> E2[YOLO Detection]
        E2 --> E3[Pose Detection]
        E3 --> E4[Ball Tracking]
        E4 --> E5[Shot Classifier]
    end

Loading

State machine: IDLE → APPROACH → RIM_CONTACT → RESOLUTION → make/miss

---

ML Pipeline

The ML pipeline is the core of the project. It combines several systems to detect and classify shots from raw video.

YOLO Object Detection — Custom YOLO26s model trained on 27,200 images with 5 detection classes (ball, human, rim, made, shoot) at 1280px resolution. A second rim-crop inference pass runs when the ball approaches the rim, giving 4–8x more pixels for critical frames.

Shot Detection — MediaPipe pose detection catches shooting stances before the ball leaves the hand. A ball-trajectory fallback trigger handles cases where pose detection misses.

Ball Tracking — BoT-SORT multi-object tracking with trajectory interpolation for frames where the ball is temporarily lost.

4-Signal Hybrid Classifier — The hardest problem: a ball going through the net and one barely missing look nearly identical to a camera. The solution combines four independent signals with camera-angle-aware weights:

1. Line Intersection — does the ball's trajectory cross through the rim opening?
2. Scoring Zone — is the ball in the net area, moving downward?
3. Trajectory Prediction — predicted arc + net occlusion detection (ball disappearing = passing through net)
4. Net Interaction — horizontal drift after exiting the rim (makes drift more due to net contact)

Each signal is weak alone. Together with veto conditions (airball detection, re-entry penalty, occlusion checks), they reach 81% classification accuracy on detected shots.

For the full classifier design with signal weights and edge-case handling, see docs/architecture.md.

---

Mobile App

Built with Expo React Native + NativeWind (Tailwind CSS).

• Auth flow — login, signup, forgot password via Supabase Auth
• Dashboard — session history, aggregate shooting stats
• Video upload — record or select from library, upload to Cloudflare R2
• Session detail — shot-by-shot breakdown with make/miss results and trajectory overlay
• Real-time progress — processing status updates while the ML worker runs

---

Project Structure

ShotTracker/
├── worker/                        # Python ML pipeline
│   ├── app/
│   │   ├── ml/
│   │   │   ├── detector.py        # YOLODetector + rim-crop inference
│   │   │   ├── tracker.py         # TrajectoryTracker + StableRimPosition
│   │   │   ├── classifier.py      # State machine + 4-signal classifier
│   │   │   ├── pose_detector.py   # MediaPipe pose detection
│   │   │   └── pipeline.py        # Main processing pipeline
│   │   ├── video/extractor.py     # FFmpeg frame extraction
│   │   └── tasks/process_video.py # Job orchestration + Supabase writes
│   ├── handler.py                 # RunPod serverless entrypoint
│   ├── Dockerfile
│   └── requirements.txt
├── mobile/                        # Expo React Native app
│   ├── app/
│   │   ├── (auth)/                # Login, signup, forgot password
│   │   ├── (tabs)/                # Dashboard, sessions, profile
│   │   ├── session/               # Session detail + trajectory view
│   │   └── new-session/           # Camera, library, upload flow
│   ├── components/                # SessionCard, StatCard, VideoUploader, etc.
│   └── hooks/                     # 10 custom hooks (auth, sessions, upload)
├── backend/                       # FastAPI REST API
│   └── app/
│       ├── routers/               # /sessions, /stats endpoints
│       ├── services/              # RunPod dispatch, R2 storage, sessions
│       └── models/                # Pydantic models
└── supabase/                      # DB migrations + auth config

---

Getting Started

Prerequisites

• Python 3.11+
• CUDA-capable GPU (recommended) or CPU
• FFmpeg

Run the ML Pipeline Locally

cd worker
python -m venv venv && source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install -r requirements.txt

# Run classifier test on a video
python test_classifier_debug.py --video /path/to/your/video.mov

Model weights are not included in the repo (large files). Contact me for access.

---

Tech Stack

ML: Python · PyTorch · YOLO26 · MediaPipe · OpenCV · BoT-SORT · FFmpeg Backend: FastAPI · RunPod Serverless GPU · Docker · Digital Ocean Storage: Cloudflare R2 (video) · Supabase (PostgreSQL + Auth) Mobile: React Native · Expo · TypeScript · NativeWind