🚁 Aerial Guardian — Drone Multi-Object Tracking Pipeline

YOLOv8n + Custom ByteTrack + Ego-Motion Compensation for VisDrone MOT

---

Quick Start

# 1. Clone and install
git clone <your-repo>
cd aerial_guardian
pip install -r requirements.txt

# 2. Download VisDrone MOT validation set and place under data/
# https://drive.google.com/file/d/1rqnKe9IgU_crMaxRoel9_nuUsMEBBVQu

# 3. Run on a single sequence
python run.py \
  --source data/VisDrone2019-MOT-val/sequences/uav0000137_00458_v \
  --output output/uav0000137_tracked.mp4

# 4. Run on a video file
python run.py --source data/drone_clip.mp4 --output output/tracked.mp4

# 5. Evaluate all sequences (produces .txt files for submission)
python evaluate.py \
  --sequences data/VisDrone2019-MOT-val/sequences \
  --output    results/

# 6. Benchmark FPS
python benchmark.py --source data/VisDrone2019-MOT-val/sequences/uav0000137_00458_v

Useful flags

Flag	Effect
--no-ego	Disable ego-motion compensation
--no-slice	Disable sliced inference (faster, less accurate)
--conf 0.3	Override detection confidence threshold
--max-frames 300	Limit to first N frames
--show	Live preview window

---

Project Structure

aerial_guardian/
├── run.py                   # Main entry point
├── benchmark.py             # Per-component FPS profiler
├── evaluate.py              # VisDrone MOT submission generator
├── config.yaml              # All tunable parameters
├── requirements.txt
└── src/
    ├── detector.py          # YOLOv8n + CLAHE + sliced inference
    ├── kalman_filter.py     # Constant-velocity Kalman filter
    ├── byte_tracker.py      # ByteTrack (custom implementation)
    ├── ego_motion.py        # ORB + RANSAC homography estimation
    ├── visualizer.py        # Bounding boxes, IDs, trajectory tails
    └── pipeline.py          # Orchestrator

---

Technical Report

1. Architecture Overview and Why These Choices

Frame (BGR)
   │
   ├── EgoMotionCompensator (ORB + RANSAC) → H  (3×3 homography)
   │
   ├── AerialDetector
   │    ├── CLAHE preprocessing           (local contrast boost)
   │    ├── Tiled 640×640 slices          (SAHI-style small-object trick)
   │    ├── YOLOv8n on each tile          (~6 MB model)
   │    └── Cross-tile NMS                (merge results)
   │
   └── ByteTracker
        ├── Kalman predict all tracks
        ├── apply H to predictions        (ego-motion compensation)
        ├── Hungarian match — high-conf   (first pass)
        ├── Hungarian match — low-conf    (second pass, ByteTrack key idea)
        └── Track lifecycle management

Base model — YOLOv8n
YOLOv8 uses a CSP-DarkNet backbone with a BiFPN-style neck (PANet) and a decoupled head that separates classification from regression. The nano variant trades accuracy for speed: 6 MB on disk, ~3.2 ms/image on a GPU, ~45 ms/image on a modern CPU. It is pretrained on COCO-2017 which includes all required classes (person, bicycle, car, motorcycle, bus, truck).

Why not EfficientDet?
EfficientDet-D0 is similarly sized but slower at CPU inference due to BiFPN's repeated depthwise separable convolutions. YOLOv8n's single-pass anchor-free design is faster and simpler to deploy.

---

2. Small Object Detection — Sliced Inference

The problem: a pedestrian 50 m below a drone at 30 m altitude subtends roughly 10–20 px in a 1920×1080 frame. After YOLOv8's 5× downsampling the feature map pixel covering this object is sub-pixel; it effectively disappears before the detection head sees it.

Solution — SAHI-style tiling:
The frame is divided into 640×640 patches with 20 % overlap, and YOLOv8 is run on each patch independently. A 15 px pedestrian in the original frame now occupies ~15 px in a tile that covers only 640 px of scene width — it's the same absolute size, but it is ~20× larger relative to the tile, well within the network's detection range. After inference, each detection is translated back to original image coordinates and cross-tile NMS removes duplicates at overlap boundaries.

Overlap: without overlap, an object straddling two tiles appears partially in each and may be detected in neither. 20 % overlap ensures that every point in the image is covered by at least one full tile (given tiles ≥ 640 px and overlap 0.2, any point within 128 px of a boundary is still covered by an adjacent tile's interior).

CLAHE preprocessing:
Drone imagery often has strong vignetting and locally dark regions (asphalt shadow, tree canopy). CLAHE (Contrast Limited Adaptive Histogram Equalisation) boosts local contrast tile-by-tile on the Y channel of YCrCb, without changing colours or causing noise amplification at boundaries. It reliably improves confidence scores for small, low-contrast objects by 5–15 %.

---

3. Handling ID Switching — Ego-Motion Compensation

The core problem:
A drone panning 30 px per frame shifts every object's image-space position by 30 px. The Kalman filter's constant-velocity model predicts each track at its previous location, now 30 px away from the actual object. With a 1920 px wide frame and object bounding boxes of 20–60 px, this 30 px error easily drops IoU below the matching threshold (0.3), causing every track to fail association and then re-acquire as a new track — a complete ID switch event for every moving frame.

Solution — background homography:

1. ORB feature extraction: extract up to 1500 ORB keypoints from the current and previous grayscale frames. ORB is rotation- and scale-invariant and runs in < 5 ms on CPU.

2. Lowe ratio matching: match descriptors with a brute-force Hamming matcher + ratio test (threshold 0.75) to reject ambiguous matches.

3. RANSAC homography: fit a perspective homography H (3×3) via RANSAC with reprojection threshold 3 px. Moving objects are geometric outliers and are automatically excluded — only static background features constrain H. This gives us the pure camera-motion transform.

4. Kalman state correction (the key step):
   After predicting each track forward with the constant-velocity model, we apply H to the predicted centre:

   [cx', cy']  =  H · [cx, cy, 1]^T  (homogeneous)
   

   This moves the prediction to where a stationary world point would appear in the current frame — i.e. it accounts for the camera's translation, rotation, and zoom. If the object itself is also moving, the residual error after association will be only the object's own motion, which the Kalman velocity estimate will absorb over subsequent frames.

5. Velocity de-biasing: we subtract the ego-motion delta (dx, dy) from the stored velocity estimate so the next frame's prediction is not double-corrected.

ByteTrack's second association pass:
Even after ego-motion compensation, small objects with low detection confidence (0.1–0.4) would be discarded by a threshold filter before reaching the tracker. ByteTrack's insight is to run a second matching pass using these low-confidence detections against unmatched tracks (after the high-confidence pass). This recovers partially-occluded or motion-blurred objects that produce weak detections, without introducing false tracks from random background noise (because they are only used to extend existing tracks, not spawn new ones).

Track buffer:
Lost tracks are retained for 30 frames before deletion. If a drone momentarily flies behind an obstacle or the object passes under heavy shadow, the track remains and can re-associate when the object reappears — without an ID change.

---

4. Edge Hardware Deployment (NVIDIA Jetson)

The following steps would adapt this pipeline to run on a Jetson Nano (472 GFLOPS), Jetson Xavier NX (21 TOPS), or similar:

Step 1 — TensorRT export

# Export YOLOv8n to TensorRT FP16 engine
yolo export model=yolov8n.pt format=engine half=True imgsz=640

This alone typically gives 3–5× speedup over PyTorch on Jetson.

Step 2 — INT8 quantisation (optional)
Use a VisDrone calibration dataset of ~500 frames to calibrate INT8. Expected throughput: ~30 FPS on Xavier NX at 640×640.

Step 3 — Reduce slice count
On constrained hardware, reduce slicing to only 4 tiles (2×2 grid) or use a smaller slice size (512×512). This trades some recall on very small objects for latency.

Step 4 — ORB on GPU
OpenCV's cv2.cuda.ORB_create() and CUDA-accelerated feature matching can offload ego-motion estimation from the CPU, freeing it for tracker bookkeeping.

Step 5 — GStreamer pipeline
Replace the cv2.VideoCapture reader with a GStreamer pipeline that uses hardware-accelerated H.264/H.265 decoding (nvv4l2decoder) to minimise CPU load from video I/O.

Model size budget:

Component	Size
YOLOv8n weights	6.2 MB
YOLOv8n TRT FP16 engine (Jetson)	~18 MB
Tracker state (Python objects)	< 1 MB
Total	< 25 MB (well under 300 MB limit)

---

5. Engineering Trade-offs

Trade-off	Decision	Reasoning
Model size vs accuracy	YOLOv8n (6 MB) over YOLOv8s (22 MB)	3× size saving, ~5% mAP loss acceptable for fast on-drone inference
Slicing speed vs recall	640×640 patches, 20% overlap	Recovers objects < 15px; disabling gives ~2.5× speedup if needed
CLAHE on/off	On by default	Near-zero cost (~2 ms), consistent recall improvement for low-contrast aerial scenes
Homography model	Perspective (8 DOF) over affine (6 DOF)	Drones tilt and zoom; pure translation/rotation (affine) is insufficient
ByteTrack vs DeepSORT	ByteTrack	No re-ID model needed → smaller, faster, no extra neural network
max_lost = 30 frames	Moderate buffer	Balances ID-switch reduction vs ghost-track accumulation

---

6. Dataset Notes

VisDrone Task 4 MOT validation set contains 35 sequences, ~14,000 total frames at 1920×1080 or 960×540, recorded from DJI drones at altitudes of 5–50 m. Annotated classes: pedestrian, people, bicycle, car, van, truck, tricycle, awning-tricycle, bus, motor.

COCO-to-VisDrone class mapping used:

COCO ID	COCO name	VisDrone equivalent
0	person	pedestrian, people
1	bicycle	bicycle
2	car	car, van
3	motorcycle	motor, tricycle
5	bus	bus
7	truck	truck

---

Measured FPS (example — CPU only, Intel i7-11th Gen)

Mode	FPS
YOLO full frame only	~21
YOLO + slicing (4 tiles)	~8
Full pipeline (slicing + ego + tracker)	~6–7
Full pipeline (no slicing)	~18–20

GPU (RTX 3060): full pipeline with slicing ≈ 28–35 FPS

---

Dependencies

ultralytics  >= 8.2.0   # YOLO + model hub
opencv-python >= 4.8.0  # video I/O, CLAHE, ORB, homography
numpy  >= 1.24.0
scipy  >= 1.11.0        # linear_sum_assignment (Hungarian)
pyyaml >= 6.0
tqdm   >= 4.65.0
torch  >= 2.0.0