Skip to content

feat: add video frame retrieval#365

Closed
raoabinav wants to merge 5 commits into
StarTrail-org:mainfrom
raoabinav:feat/video-frame-retrieval
Closed

feat: add video frame retrieval#365
raoabinav wants to merge 5 commits into
StarTrail-org:mainfrom
raoabinav:feat/video-frame-retrieval

Conversation

@raoabinav

Copy link
Copy Markdown
Contributor

Evaluator Summary

  • Abi added frame-level video retrieval by sampling local video frames, embedding them with CLIP, and indexing each sampled frame as a searchable LEANN passage.
  • Design choice: each frame gets a stable video-frame-<hash> ID plus timestamp, frame index, frame size, duration, source path, media type, and model metadata, so search results point back into the original video.
  • The implementation builds from precomputed CLIP frame arrays and disables query-time recomputation for frame vectors, matching the image-retrieval design while keeping visual retrieval separate from audio transcription.
  • Quality bar: 9 targeted tests mock decoding and model inference to avoid codec/model downloads while covering sampling, metadata, stable IDs, and retrieval wiring, plus lint, format, type, lockfile, and whitespace checks.

@yichuan-w

Copy link
Copy Markdown
Collaborator

I don't think CLIP is a very good embedding model to use nowadays, especially for video, maybe Qwen3VL embedding is better

@raoabinav

Copy link
Copy Markdown
Contributor Author

Parking this PR for now to keep the review queue focused on #347. I’ll revisit/reopen this work later as a smaller clean PR once the content-hash PR lands.

@raoabinav raoabinav closed this Jun 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants