feat(tube-multiscale-fusion): two-branch multiscale temporal smoke classifier

[rensortino]

Summary

New experiment under experiments/temporal-models/tube-multiscale-fusion/ — a
two-branch temporal smoke classifier that pairs a global DINOv2 sequence
context branch with a local spatio-temporal tube transformer, fused via
cross-attention.

Motivation

Smoke and its hardest distractors (clouds, fog, haze, dust) look near-identical
in any single frame; what separates them is how they move, and they move
differently at different scales:

• Global branch (low-frequency, long time): DINOv2 embeds each of 16 frames;
  a small transformer aggregates them into one context vector capturing the
  overall shape evolution — static/drifting (cloud, fog) vs. growing/rising
  (plume).
• Local branch (high-frequency, short time): each 224×224 bbox patch is
  decomposed into a grid of spatial cells tracked over short, overlapping
  4-frame windows ("tubes"). A per-tube video transformer reads the local motion
  signature — turbulent/high-variance for smoke, smooth/coherent for fog/cloud —
  that a single global vector smooths away.
• Fusion: local tubes self-attend, then the global vector acts as the
  query in cross-attention over the tubes, weighting high-frequency local
  evidence with long-range context.

Architecture

patches (B, 16, 3, 224, 224) + mask
   ├─ Global: DINOv2 ViT-S/14 per frame → transformer aggregator → (B, 384)
   └─ Local:  extract_tubes (grid × overlapping windows) → tubelet Conv3d +
              self-attn transformer per tube → (B, N_tubes, 384) + validity mask
   → Fusion (self-attn on tubes + cross-attn global=Q/locals=KV) → (B, 384)
   → MLP head → logit

Default geometry (winner of the included resolution sweep): 2×2 grid of
112×112 cells × 4-frame windows at stride 2 → 28 tubes/seq.

Results (val, 280 tubes, single seed)

variant	grid / len / stride	tubes/seq	val F1	val acc	val prec	val rec	val PR-AUC
default (2×2)	2×2 / 4 / 2	28	0.9783	0.9786	0.9574	1.0000	0.9936
spatial_4x4	4×4 / 4 / 2	112	0.9673	0.9679	0.9500	0.9852	0.9874
spatial_8x8	8×8 / 4 / 2	448	0.9745	0.9750	0.9571	0.9926	0.9942
temporal_stride1	2×2 / 4 / 1	52	0.9781	0.9786	0.9640	0.9926	0.9954
temporal_len8	2×2 / 8 / 4	12	0.9744	0.9750	0.9638	0.9852	0.9939

Confusion matrix (default 2×2): TN=139, FP=6, FN=0, TP=135.

Benchmarking (GUIDELINES.md metrics, RTX 4090 / 24-thread CPU)

Metric	Value
Recall @ FPR = 1% / 5% / 10%	0.874 / 1.000 / 1.000
Time-to-detection (median)	2 frames ≈ 30 s after tube start (135/135 eventually fire)
Inference latency (GPU)	10.7 ms/seq → 0.67 ms/frame
Inference latency (CPU)	290 ms/seq → 18.2 ms/frame
Model size	36.4 M params (16.6 M trainable, 19.9 M frozen DINOv2)
FLOPs	226.5 GFLOPs/seq → 14.2 GFLOPs/frame

What's included

• src/tube_multiscale_fusion/ — modular global_branch, local_branch,
  fusion, classifier, and the LitTubeMultiscaleClassifier Lightning module.
  Upstream tube-building / patch-cropping primitives reused from the parent.
• scripts/ — train, evaluate, benchmark, and package_model
  (bundles checkpoint + exact params + manifest into a portable .zip).
• DVC pipeline — prepare → truncate → build_tubes → build_model_input → train → evaluate → package → benchmark, plus a resolution-sweep matrix.
• Tests — 56 unit tests (shape, masking, tube extraction, gradient flow,
  fusion); 2 @slow tests behind a marker for the real DINOv2 download.
• README — architecture diagram, motivation, sweep results, benchmarking,
  and reproduction steps.

How to reproduce

cd experiments/temporal-models/tube-multiscale-fusion
uv sync
# import data (pinned pyro-dataset v2.2.0) — see README "Data"
uv run dvc repro

Notes for reviewers

• Data: data/01_raw/datasets_full/ is gitignored; on a clean checkout it
  comes from the dvc import commands in the README (pinned to v2.2.0), not
  from the DVC remote.
• DVC remote: s3://pyro-vision-rd/dvc/experiments/tube-multiscale-fusion/.
  Artifacts (checkpoints, model package, reports) are pushed via dvc push;
  benchmark.json is a cache: false metric committed directly to git.
• Single-seed results; no multi-seed CI run yet. The small val set (280) means
  early-stopping on val/f1 saturates quickly — see README for the schedule.

[Chouffe]

Really excited about this approach 🥳 — explicitly modelling the spatio-temporal signal at two scales (global low-frequency sequence context + local high-frequency tube motion) is exactly the kind of "how does it move" cue that should separate smoke from clouds/fog/haze, and the cross-attention fusion is a clean way to weight the two. Great direction!

Really nice PR overall too 👏🏿 .
Clean modular split (global_branch / local_branch / fusion / classifier / lit_module), genuinely strong test coverage, committed reproducible metrics, and it follows repo conventions.

Findings below are mostly polish — one CI blocker.

🔴 Blocker — ruff format --check fails (CI will go red)

With the locked ruff 0.15.7, four files need reformatting:

Would reformat: scripts/benchmark.py
Would reformat: scripts/evaluate.py
Would reformat: tests/test_classifier.py
Would reformat: tests/test_local_branch.py

Fix:

make format

🟠 Should fix

1. Dead prepare stage + unused heavy deps. scripts/prepare.py downloads YOLO weights into data/01_raw/models, but nothing consumes them — build_tubes.py reads label .txt files directly and its docstring says "No YOLO inference is performed." No DVC stage depends on the prepare output. Correspondingly ultralytics, opencv-python-headless, pandas, and pydantic are declared in pyproject.toml but never imported anywhere in src/, scripts/, or tests/ (ultralytics especially is a heavy footprint). Suggest dropping the prepare stage + these deps, or wiring the detector in if it's actually intended.

2. Tube-building algo is now in the lib — this experiment forks it. The canonical tube-building implementation now lives in lib/bbox-tube-temporal/ (package bbox-tube-temporal-core): compute_iou, match_detections, build_tubes, interpolate_gaps, select_longest_tube, tube_from_record — exactly the API re-implemented here in src/tube_multiscale_fusion/tubes.py (and data.py / model_input.py / types.py look similarly duplicated). The lib version is also ahead (e.g. merge_colocated_tubes). Per "shared code goes under lib/ with proper packaging," this experiment should depend on bbox-tube-temporal-core and import from it rather than carry a fork that will drift.

3. No pyrocore integration. pyrocore is a declared dependency (with a [tool.uv.sources] path) but is never imported. The convention is for each experiment to expose a TemporalModel subclass so it plugs into the shared eval interface / temporal-model-explorer — without an adapter this model can't be compared alongside the others. At minimum worth a tracked follow-up; ideally a thin wrapper around LitTubeMultiscaleClassifier.

🟡 Minor / nits

• Copy-paste docstrings referencing the wrong architecture:
  • tubes.py module docstring mentions "the LSTM's need for temporally-ordered features" — there's no LSTM.
  • augment.py TemporalTubeTransform mentions "pack_padded_sequence (used by the GRU head)" — no GRU / pack_padded_sequence. The prefix-compaction is correct and still needed; just the stated rationale is stale.
• Dataset version — inconsistent + now outdated: PR body & README say pinned pyro-dataset v2.2.0, but data.py's list_sequences docstring says "nested pyro-dataset v3.0.0 layout." Which is authoritative? Also note pyro-dataset v4.0.0 is now available — worth deciding whether to adopt it (results would need a re-run) or at least documenting why this experiment pins an older version.

💡 Design suggestions (follow-up, not blocking)

The architecture is great but feels heavier than the current benchmark justifies. Three ideas, in priority order, for probing whether the spatial encoding can be much simpler:

1. Ablate the local branch before optimizing it. This is the highest-value experiment, and it's nearly free — you already have the global branch. Run global-branch-only (DINOv2 + temporal aggregator → head, no local branch, no fusion) on the same split. Two signals say the local branch may not be earning its complexity: the metrics are already near-saturated (F1 0.978, recall@FPR=5% = 1.0, 0 false negatives) on a small single-seed val set, and the resolution sweep shows 2×2 / 28 tubes beat 8×8 / 448 tubes — i.e. adding spatial granularity hurt, the opposite of what you'd expect if fine local structure were the discriminative signal. If global-only lands within noise of the full model, the entire local branch + fusion is removable. If it doesn't, inspect which sequences global-only gets wrong (presumably the fog/cloud/haze distractors) and let those concrete failure cases dictate the minimal local encoder — rather than carrying the full tube transformer on the assumption it's needed.

2. If local motion does help, reuse the DINOv2 patch tokens instead of a second encoder. The global branch already runs DINOv2 on every frame but keeps only the CLS token (global_pool="token") and throws away the patch-token grid (16×16 for ViT-S/14 @ 224). That grid is a spatial decomposition — for free, and at far better quality than a small Conv3d+transformer trained from scratch on a tiny dataset. Concretely: switch the backbone to return patch tokens (B, T, 256, 384), then capture local motion with a cheap temporal op per patch location — e.g. temporal std-dev/variance over the T axis (directly encodes "turbulent smoke vs smooth fog"), or a tiny shared temporal MLP/attention per location. This deletes local_branch.py's tube extraction and the tubelet encoder while keeping the multi-scale intuition, and it removes the awkward fact that the model currently encodes the same pixels twice (once via DINOv2, once via the raw-pixel tube encoder). The 2×2-beats-8×8 result also suggests you can pool that patch grid down aggressively without losing anything.

3. Replace cross-attention fusion with pooling + concat. Independent of the above, a single global query cross-attending ~28 tubes (FusionModule, two layers of self-attn + cross-attn + FFN) is a lot of machinery for "summarize the local evidence and combine it with the global vector." Mean- and/or max-pool the local embeddings over the (masked) tube axis and concat([global_vec, pooled_local]) → MLP head is a strong, much smaller baseline — and worth running as its own ablation against the cross-attention version to confirm the attention is actually buying accuracy. This drops fusion.py entirely.

Net "simplest viable" to try alongside the current model: global DINOv2 sequence branch + temporal-variance pooling over its own patch-token grid + concat → MLP — same two-scale story, but no tube extraction, no Conv3d encoder, no cross-attention.