Skip to content

Commit b38dc21

Browse files
author
Abdel Darwish
committed
Add VLM-based semantic footage rating tools
Adds four tools that give video libraries semantic understanding with a local vision-language model (Ollama served, e.g. Gemma 4 or Qwen-VL). This covers what static CLIP retrieval cannot: temporal and behavioral semantics. What is included: - vlm_clip_rating: coarse rating of a clip folder (behavior, camera quality, composition, subject visibility, segments, highlights) - vlm_zoom_rating: frame-accurate sub-beat timestamps and deep-dive descriptions for flagged windows - vlm_editorial_ranking: composite scores, leaderboards, and match-cut chains built from the ratings - vlm_comparative_rank: relative ranking of candidate clips with editorial reasoning (shows 4 clips at once for calibrated scores) - vlm-footage-rating skill documenting the workflow - Unit tests for all four tools (Ollama HTTP mocked, no model needed) All stages are idempotent and resume from JSONL output, so re-running after adding footage only processes new clips. Fully local, no API keys.
1 parent 4eab34c commit b38dc21

10 files changed

Lines changed: 2451 additions & 0 deletions

File tree

‎.agents/skills/vlm-footage-rating/SKILL.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
---
2+
name: vlm-footage-rating
3+
description: Semantic video understanding for footage selection. Use when a clip library needs to be rated by behavior, camera stability, subject visibility, composition, or temporal structure; when clips must be selected for a montage by semantics instead of filename; when the editor needs frame-accurate timestamps of interesting moments; or when two candidate clips need a relative comparison with reasoning. Works fully local with an Ollama vision model (e.g. Gemma 4 12B, Gemma 3n, Qwen-VL). No API keys required.
4+
license: MIT
5+
metadata:
6+
requires:
7+
env: []
8+
ollama: true
9+
---
10+
11+
# VLM Footage Rating (Semantic Video Understanding)
12+
13+
Rate a folder of video clips with a local vision-language model and get an
14+
editorial database: behavior, camera quality, composition, subject (product)
15+
visibility, timestamped segments, highlights, rankings, and match-cut
16+
continuity. This is the layer static CLIP retrieval cannot provide:
17+
temporal and behavioral semantics.
18+
19+
## Pipeline Overview
20+
21+
```
22+
vlm_clip_rating -> clip_tags.jsonl (coarse: behavior/camera/shot/subject/segments)
23+
vlm_zoom_rating -> clip_zooms.jsonl (frame-accurate timestamps + deep-dive descriptions)
24+
vlm_editorial_ranking -> editorial_rankings.json (composite scores, leaderboards, match cuts)
25+
vlm_comparative_rank -> comparative_rankings.jsonl (optional: relative ranking with reasoning)
26+
```
27+
28+
Each stage is idempotent (skips already-processed clips), so re-running after
29+
adding footage only processes the new clips.
30+
31+
## Prerequisites
32+
33+
- ffmpeg/ffprobe on PATH
34+
- Ollama running with a vision model: `ollama pull gemma4:12b` (12B, ~8GB
35+
VRAM) or `gemma3n:4b` for smaller GPUs
36+
37+
## Usage
38+
39+
### 1. Coarse rating
40+
41+
```python
42+
from tools.video.vlm_clip_rating import VlmClipRating
43+
tool = VlmClipRating()
44+
tool.execute({
45+
"input_dir": "/path/to/clips",
46+
"output_path": "/path/to/clip_tags.jsonl",
47+
"focus_prompt": "a black collar (the product being advertised)",
48+
"model": "gemma4:12b",
49+
})
50+
```
51+
52+
`focus_prompt` is optional: set it to whatever subject the edit cares about
53+
(a product, an animal behavior, a person) and the model rates its visibility
54+
in every clip. Leave empty for generic footage rating.
55+
56+
### 2. Zoom pass (frame-accurate timestamps)
57+
58+
```python
59+
from tools.video.vlm_zoom_rating import VlmZoomRating
60+
VlmZoomRating().execute({
61+
"ratings_path": "/path/to/clip_tags.jsonl",
62+
"output_path": "/path/to/clip_zooms.jsonl",
63+
})
64+
```
65+
66+
Re-examines each flagged highlight/segment at 4 fps, producing sub-beats with
67+
precise start/end times, camera angle, subject facing direction (for match
68+
cuts), deep-dive descriptions, and vibe.
69+
70+
### 3. Editorial ranking
71+
72+
```python
73+
from tools.video.vlm_editorial_ranking import VlmEditorialRanking
74+
VlmEditorialRanking().execute({
75+
"ratings_path": "/path/to/clip_tags.jsonl",
76+
"zooms_path": "/path/to/clip_zooms.jsonl", # optional
77+
"output_path": "/path/to/editorial_rankings.json",
78+
"weights": {"stability": 0.25, "quality": 0.25, "subject": 0.25,
79+
"composition": 0.15, "vibe": 0.10},
80+
})
81+
```
82+
83+
Weights must sum to 1.0 and are campaign-tunable (bias toward product shots,
84+
stability, or energy).
85+
86+
### 4. Comparative ranking (optional tiebreaker)
87+
88+
```python
89+
from tools.video.vlm_comparative_rank import VlmComparativeRank
90+
VlmComparativeRank().execute({
91+
"rankings_path": "/path/to/editorial_rankings.json",
92+
"output_path": "/path/to/comparative_rankings.jsonl",
93+
"purpose": "subject_hero",
94+
})
95+
```
96+
97+
Shows 4 candidate clips in one context, asks for a relative ranking,
98+
calibrated scores, and reasons for best/worst. Use when two clips score
99+
close and you want the model to argue about which wins.
100+
101+
## Output Schema Highlights
102+
103+
| Field | Meaning |
104+
|---|---|
105+
| `overall.behavior` | walking_calm, pulling, sniffing, trotting, sitting, lying, greeting, playing, expression, other |
106+
| `overall.energy` | calm, neutral, excited, hyper |
107+
| `camera.stability_score` | 0-1 camera shake quality |
108+
| `shot.type` | extreme_wide ... extreme_close_up |
109+
| `shot.rule_of_thirds_score` | 0-1 composition |
110+
| `product.subject_visibility` | not_visible, partial, clear, featured |
111+
| `product.subject_quality_score` | 0-1 how well the subject of interest is shown |
112+
| `segments[].start_s/end_s` | coarse timestamped beats |
113+
| `highlights[].start_s/end_s` | keeper moments (coarse) |
114+
| zoom `sub_beats[]` | frame-accurate beats: start_s/end_s, camera_angle, subject_facing, deep_dive, vibe, use |
115+
116+
## Editing Recipes
117+
118+
- **Subject hero shots**: filter `product.subject_visibility` in
119+
("featured", "clear"), rank by `subject_quality_score`.
120+
- **Stability gate**: only use clips with `camera.stability_score >= 0.9`.
121+
- **Match cuts**: `editorial_rankings.json` -> `match_cuts` -> chains by
122+
subject facing direction (left/right). Cut same-direction shots together.
123+
- **Story beats**: pick behavior buckets (open wide, calm walk, action,
124+
subject close-up) and use zoom timestamps for precise cut points.
125+
126+
## Pitfalls
127+
128+
- The VLM occasionally emits malformed JSON (string entries in arrays,
129+
non-numeric timestamps). The tools guard this; if you hand-parse the
130+
JSONL, filter `isinstance(x, dict)` and use safe float conversion.
131+
- Zoom sub-beat timestamps are window-relative: real clip time is
132+
`window_start_s + sub_beat.start_s`.
133+
- First clip per run is slower (model load). Subsequent clips are ~5-15s.
134+
- Runtime scales with clip count: ~8-15s per clip for coarse, ~30s per
135+
zoom window. Budget accordingly for large libraries.

‎.claude/skills/vlm-footage-rating/SKILL.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
---
2+
name: vlm-footage-rating
3+
description: Semantic video understanding for footage selection. Use when a clip library needs to be rated by behavior, camera stability, subject visibility, composition, or temporal structure; when clips must be selected for a montage by semantics instead of filename; when the editor needs frame-accurate timestamps of interesting moments; or when two candidate clips need a relative comparison with reasoning. Works fully local with an Ollama vision model (e.g. Gemma 4 12B, Gemma 3n, Qwen-VL). No API keys required.
4+
license: MIT
5+
metadata:
6+
requires:
7+
env: []
8+
ollama: true
9+
---
10+
11+
# VLM Footage Rating (Semantic Video Understanding)
12+
13+
Rate a folder of video clips with a local vision-language model and get an
14+
editorial database: behavior, camera quality, composition, subject (product)
15+
visibility, timestamped segments, highlights, rankings, and match-cut
16+
continuity. This is the layer static CLIP retrieval cannot provide:
17+
temporal and behavioral semantics.
18+
19+
## Pipeline Overview
20+
21+
```
22+
vlm_clip_rating -> clip_tags.jsonl (coarse: behavior/camera/shot/subject/segments)
23+
vlm_zoom_rating -> clip_zooms.jsonl (frame-accurate timestamps + deep-dive descriptions)
24+
vlm_editorial_ranking -> editorial_rankings.json (composite scores, leaderboards, match cuts)
25+
vlm_comparative_rank -> comparative_rankings.jsonl (optional: relative ranking with reasoning)
26+
```
27+
28+
Each stage is idempotent (skips already-processed clips), so re-running after
29+
adding footage only processes the new clips.
30+
31+
## Prerequisites
32+
33+
- ffmpeg/ffprobe on PATH
34+
- Ollama running with a vision model: `ollama pull gemma4:12b` (12B, ~8GB
35+
VRAM) or `gemma3n:4b` for smaller GPUs
36+
37+
## Usage
38+
39+
### 1. Coarse rating
40+
41+
```python
42+
from tools.video.vlm_clip_rating import VlmClipRating
43+
tool = VlmClipRating()
44+
tool.execute({
45+
"input_dir": "/path/to/clips",
46+
"output_path": "/path/to/clip_tags.jsonl",
47+
"focus_prompt": "a black collar (the product being advertised)",
48+
"model": "gemma4:12b",
49+
})
50+
```
51+
52+
`focus_prompt` is optional: set it to whatever subject the edit cares about
53+
(a product, an animal behavior, a person) and the model rates its visibility
54+
in every clip. Leave empty for generic footage rating.
55+
56+
### 2. Zoom pass (frame-accurate timestamps)
57+
58+
```python
59+
from tools.video.vlm_zoom_rating import VlmZoomRating
60+
VlmZoomRating().execute({
61+
"ratings_path": "/path/to/clip_tags.jsonl",
62+
"output_path": "/path/to/clip_zooms.jsonl",
63+
})
64+
```
65+
66+
Re-examines each flagged highlight/segment at 4 fps, producing sub-beats with
67+
precise start/end times, camera angle, subject facing direction (for match
68+
cuts), deep-dive descriptions, and vibe.
69+
70+
### 3. Editorial ranking
71+
72+
```python
73+
from tools.video.vlm_editorial_ranking import VlmEditorialRanking
74+
VlmEditorialRanking().execute({
75+
"ratings_path": "/path/to/clip_tags.jsonl",
76+
"zooms_path": "/path/to/clip_zooms.jsonl", # optional
77+
"output_path": "/path/to/editorial_rankings.json",
78+
"weights": {"stability": 0.25, "quality": 0.25, "subject": 0.25,
79+
"composition": 0.15, "vibe": 0.10},
80+
})
81+
```
82+
83+
Weights must sum to 1.0 and are campaign-tunable (bias toward product shots,
84+
stability, or energy).
85+
86+
### 4. Comparative ranking (optional tiebreaker)
87+
88+
```python
89+
from tools.video.vlm_comparative_rank import VlmComparativeRank
90+
VlmComparativeRank().execute({
91+
"rankings_path": "/path/to/editorial_rankings.json",
92+
"output_path": "/path/to/comparative_rankings.jsonl",
93+
"purpose": "subject_hero",
94+
})
95+
```
96+
97+
Shows 4 candidate clips in one context, asks for a relative ranking,
98+
calibrated scores, and reasons for best/worst. Use when two clips score
99+
close and you want the model to argue about which wins.
100+
101+
## Output Schema Highlights
102+
103+
| Field | Meaning |
104+
|---|---|
105+
| `overall.behavior` | walking_calm, pulling, sniffing, trotting, sitting, lying, greeting, playing, expression, other |
106+
| `overall.energy` | calm, neutral, excited, hyper |
107+
| `camera.stability_score` | 0-1 camera shake quality |
108+
| `shot.type` | extreme_wide ... extreme_close_up |
109+
| `shot.rule_of_thirds_score` | 0-1 composition |
110+
| `product.subject_visibility` | not_visible, partial, clear, featured |
111+
| `product.subject_quality_score` | 0-1 how well the subject of interest is shown |
112+
| `segments[].start_s/end_s` | coarse timestamped beats |
113+
| `highlights[].start_s/end_s` | keeper moments (coarse) |
114+
| zoom `sub_beats[]` | frame-accurate beats: start_s/end_s, camera_angle, subject_facing, deep_dive, vibe, use |
115+
116+
## Editing Recipes
117+
118+
- **Subject hero shots**: filter `product.subject_visibility` in
119+
("featured", "clear"), rank by `subject_quality_score`.
120+
- **Stability gate**: only use clips with `camera.stability_score >= 0.9`.
121+
- **Match cuts**: `editorial_rankings.json` -> `match_cuts` -> chains by
122+
subject facing direction (left/right). Cut same-direction shots together.
123+
- **Story beats**: pick behavior buckets (open wide, calm walk, action,
124+
subject close-up) and use zoom timestamps for precise cut points.
125+
126+
## Pitfalls
127+
128+
- The VLM occasionally emits malformed JSON (string entries in arrays,
129+
non-numeric timestamps). The tools guard this; if you hand-parse the
130+
JSONL, filter `isinstance(x, dict)` and use safe float conversion.
131+
- Zoom sub-beat timestamps are window-relative: real clip time is
132+
`window_start_s + sub_beat.start_s`.
133+
- First clip per run is slower (model load). Subsequent clips are ~5-15s.
134+
- Runtime scales with clip count: ~8-15s per clip for coarse, ~30s per
135+
zoom window. Budget accordingly for large libraries.

‎skills/INDEX.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -284,6 +284,7 @@ Cross-cutting skills that apply to all pipelines:
284284
| Checkpoint Protocol | `meta/checkpoint-protocol.md` | When/how to checkpoint and request human approval |
285285
| Skill Creator | `meta/skill-creator.md` | Dynamically create new skills during pipeline runs |
286286
| Animation Runtime Selector | `meta/animation-runtime-selector.md` | Choose render runtime + animation library per scene |
287+
| Buzzback Footage Production | `meta/buzzback-production.md` | Cut dog-walk footage from AI ratings (behavior/collar/stability/match-cuts) |
287288
| Taste Direction | `meta/taste-direction.md` | Convert a brief into taste dials, anti-patterns, and reference strategy for proposal/playbook/atelier work |
288289
| Bespoke Composition (Atelier) | `meta/bespoke-composition.md` | Hand-author a composition from scratch (hero work) — no stock scene-types; routes art-direction → motion principles → engine mechanics → atelier render |
289290

@@ -320,3 +321,4 @@ Claude Code accesses them via symlinks in `.claude/skills/`.
320321
| **AI Video/Image/TTS/Avatar (Kling Official)** | `kling-official` - official direct API auth, Classic/Turbo/Omni task protocols, multi-reference Omni syntax, internal Elements/Account Usage helpers, callback notes, TTS voice parameters, avatar/lip-sync face selection, error handling, and cost governance for `kling_official_video` / `kling_official_image` / `kling_tts` / `kling_avatar` / `kling_lip_sync` | Local OpenMontage skill |
321322
| **AI Video (Premium)** | `seedance-2-0` — preferred premium default (cinematic, trailer, multi-shot, lip-sync, synced audio); accessed via `seedance_video` (fal.ai) or `heygen_video` Avatar Shots | Local OpenMontage skill |
322323
| **Infrastructure** | `acestep`, `ltx2`, `playwright-recording` | `digitalsamba/claude-code-video-toolkit` |
324+
| **Video Understanding** | `vlm-footage-rating` | Local OpenMontage skill (VLM clip rating, zoom timestamps, editorial ranking) |

0 commit comments

Comments
 (0)