feat(video): add VideoPipeline, VideoFormatOption, and InputFormat.VIDEO#3768

Open

Itsxenonli wants to merge 42 commits into

docling-project:mainfrom

Itsxenonli:feat/video-pipeline

Itsxenonli commented Jul 7, 2026

Contributor

Add InputFormat.VIDEO with extensions [mp4, avi, mov, mkv, webm] and video MIME types, moved out of InputFormat.AUDIO. Add VideoFormatOption and register it in DocumentConverter. Add VideoPipeline that extracts audio via ffmpeg, transcribes via the shared ASR transcriber, samples frames using FixedIntervalFrameSampler or SimpleSceneChangeFrameSampler, and merges transcript + frames by timestamp into a DoclingDocument. Add VideoFrameSamplingMode enum and VideoPipelineOptions.

33 routing tests pass.

Checklist:

Documentation has been updated, if necessary.
Examples have been added, if necessary.
Tests have been added, if necessary.


          feat(video): add VideoPipeline, VideoFormatOption, and InputFormat.VIDEO

6ba4686

Add InputFormat.VIDEO with extensions [mp4, avi, mov, mkv, webm] and
video MIME types, moved out of InputFormat.AUDIO. Add VideoFormatOption
and register it in DocumentConverter. Add VideoPipeline that extracts
audio via ffmpeg, transcribes via the shared ASR transcriber, samples
frames using FixedIntervalFrameSampler or SimpleSceneChangeFrameSampler,
and merges transcript + frames by timestamp into a DoclingDocument.
Add VideoFrameSamplingMode enum and VideoPipelineOptions.

33 routing tests pass.

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>

Itsxenonli marked this pull request as draft

July 7, 2026 08:51

github-actions Bot commented Jul 7, 2026

Contributor

✅ DCO Check Passed

Thanks @Itsxenonli, all your commits are properly signed off. 🎉

mergify Bot commented Jul 7, 2026

Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

codecov Bot commented Jul 7, 2026 •

edited

Loading

Codecov Report

❌ Patch coverage is 27.80749% with 270 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
docling/pipeline/video_pipeline.py	23.35%	105 Missing ⚠️
docling/utils/speaker_diarization.py	21.62%	87 Missing ⚠️
docling/utils/video_frame_sampling.py	18.18%	63 Missing ⚠️
docling/cli/main.py	0.00%	15 Missing ⚠️

📢 Thoughts on this report? Let us know!

Itsxenonli added 6 commits

July 7, 2026 11:06


          fix(video): add epsilon to frame TrackSource end_time to satisfy vali…

1c3af23

…dation

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): add epsilon to frame TrackSource end_time and add export_…

c053005

…video_document_to_html

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): add export_video_document_to_html utility

c5be21d

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): replace threshold with adaptive local peak detection and…

777fe6d

… sharpness scoring

Replace SimpleSceneChangeFrameSampler fixed threshold with scipy
find_peaks prominence-based local peak detection. Self-calibrates per
video, no manual threshold config needed. Add sharpness scoring via
Laplacian variance to pick the sharpest keyframe per scene, avoiding
motion-blurred frames during pans. Update VideoPipelineOptions to use
scene_change_prominence instead of scene_change_threshold.

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): reduce default smooth_window from 5 to 2

83b5f0d

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): remove smoothing by setting smooth_window=1

da6e61a

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>

PeterStaar-IBM requested review from PeterStaar-IBM and dolfim-ibm

July 7, 2026 15:20

Itsxenonli added 5 commits

July 8, 2026 10:12


          feat(video): add cuts_per_minute mode alongside prominence-based dete…

1d49089

…ction

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): move html import to top of file

a58903a

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): make scene_change_prominence optional (None = auto)

eaa4504

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          style: ruff format pipeline_options.py

44ee120

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(test): update audio extensions test to reflect video format split

e49af4e

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>

Itsxenonli marked this pull request as ready for review

July 8, 2026 11:23

Itsxenonli added 5 commits

July 8, 2026 14:37


          feat(video): hook video HTML export into CLI --to html output

53c6146

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): add video CLI flags --video-sampling-mode, --video-frame…

195d7cf

…-interval, --video-cuts-per-minute, --video-prominence

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): import VideoFormatOption in CLI convert function

21b06a5

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          docs(video): add recommended configs for meetings and lectures

ae981f8

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): require explicit video config, show helpful error with -…

f298e02

…-help hint

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>

Itsxenonli force-pushed the feat/video-pipeline branch from 2e492b2 to f298e02 Compare

July 8, 2026 13:43

Itsxenonli added 5 commits

July 9, 2026 09:48


          feat(video): add speaker diarization via resemblyzer with auto speake…

713daec

…r count

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(diarization): remove deprecated VoiceEncoder.params attribute

8ad47e0

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(diarization): finer window step and extend last segment to audio end

795781f

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): merge Whisper segments into sentences before diarization

29b7906

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          feat(video): remove smoothing, clean up scene detection

a9533bc

Signed-off-by: Nikolaos Bekas <bekas.nikolaoszug@gmail.com>

Itsxenonli added 6 commits

July 9, 2026 08:32


          feat(video): make speaker diarization optional via --video-diarizatio…

0d6b6f2

…n flag

Signed-off-by: Nikolaos Bekas <bekas.nikolaoszug@gmail.com>


          fix(video): use self.pipeline_options.enable_diarization

6fe8d51

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): fix diarization try/except indentation

90e9e00

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): add video_diarization CLI parameter definition

7a8e375

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(diarization): use subprocess isolation for preprocess_wav to avoi…

…d webrtcvad deadlock after Whisper

Signed-off-by: Nikolaos Bekas <bekas.nikolaoszug@gmail.com>


          fix(cli): only raise video config error when --from video is explicit…

b5f5fa8

…ly set

Signed-off-by: Nikolaos Bekas <bekas.nikolaoszug@gmail.com>

PeterStaar-IBM requested changes

View reviewed changes

docling/cli/main.py Outdated

docling/pipeline/asr_transcriber.py Outdated

docling/pipeline/video_pipeline.py Outdated

docling/utils/speaker_diarization.py

Itsxenonli added 9 commits

July 10, 2026 11:09


          fix(cli): use standard save_as_html for video output, remove custom e…

cbadc59

…xporter

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          refactor(asr): make merge_into_sentences private (_merge_into_sentences)

df5bf30

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          refactor(diarization): make SpeakerSegment and DiarizationResult private

0250fba

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          refactor(video): remove HTML export utility from video_pipeline (hand…

65d7ffd

…led by docling-core serializer)

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix(video): fix _DiarizationResult import after rename

924cedb

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          fix: ruff format and fix remaining DiarizationResult reference

266bee6

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          style: ruff format cli/main.py and video_pipeline.py

3e99c63

Signed-off-by: itsxenonli <bekas.nikolaoszug@gmail.com>


          docs(examples): add video pipeline example notebook

48414ea

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>


          style: ruff format video_pipeline notebook

c7d115f

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>

ceberam requested changes

View reviewed changes

docs/examples/video_pipeline.ipynb

docling/datamodel/pipeline_options.py Outdated

docling/datamodel/pipeline_options.py Outdated

docling/utils/speaker_diarization.py Outdated

docling/pipeline/video_pipeline.py Outdated

docling/utils/speaker_diarization.py Outdated

docling/utils/speaker_diarization.py Outdated

docling/utils/video_frame_sampling.py

tests/test_video_format_routing.py

Comment on lines +28 to +30

+              def test_input_format_video_exists():
+                  assert hasattr(InputFormat, "VIDEO")
+                  assert InputFormat.VIDEO.value == "video"

ceberam Jul 14, 2026

Member

This is a trivially self-validating test. Some more meaningful routing tests, like the MIME parametrized checks, are fine but the trivial ones, like this one, should be removed.

docling/cli/main.py Outdated

Itsxenonli and others added 5 commits

July 14, 2026 15:19


          Apply suggestion from @ceberam

355fd75

Co-authored-by: Cesar Berrospi Ramis <75900930+ceberam@users.noreply.github.com>
Signed-off-by: Nikolaos Bekas <bekas.nikolaoszug@gmail.com>


          fix: video pipeline options

7b07099

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>


          fix: video pipeline options

ce85ed9

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>


          fix: video pipeline options

f1e1738

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>


          fix: video pipeline options

5a2ae0f

Signed-off-by: Itsxenonli <bekas.nikolaoszug@gmail.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet