Skip to content

Commit c130ee0

Browse files
AdnanElAssadi56claudeSamoed
authored
Fix bidirlm omni double frame sampling (#4700)
* fix: disable inner video sampler in BidirLM-Omni encoder VideoCollator already samples frames at the requested fps, then drops the video metadata before handing a 4D tensor to the model. With do_sample_frames=True (the BidirLM-Omni video_preprocessor_config default), Qwen3VLVideoProcessor.sample_frames hits the metadata.fps=24 fallback and downsamples to ~T/12, throwing away most of the frames the outer collator extracted. Set do_sample_frames=False on the loaded processor at construction so sampling happens exactly once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * update method for setting frames sampling * simplify * move imports inside --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com>
1 parent 3bfe4aa commit c130ee0

1 file changed

Lines changed: 17 additions & 0 deletions

File tree

mteb/models/model_implementations/bidirlm_omni_models.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,23 @@ def __init__(
9494
max_samples: int | None = None,
9595
**kwargs: Any,
9696
) -> None:
97+
from transformers import AutoVideoProcessor
98+
99+
processor_kwargs = kwargs.get("processor_kwargs", {})
100+
101+
# VideoCollator already samples frames; skip inner sampling.
102+
video_processor = AutoVideoProcessor.from_pretrained(
103+
model_name,
104+
revision=revision,
105+
trust_remote_code=True,
106+
do_sample_frames=False,
107+
**kwargs,
108+
)
109+
processor_kwargs |= {
110+
"video_processor": video_processor,
111+
}
112+
113+
kwargs["processor_kwargs"] = processor_kwargs
97114
self.model = SentenceTransformer(
98115
model_name,
99116
revision=revision,

0 commit comments

Comments
 (0)