feat: add OpenCV inpaint fallback for videos where auto-detection fails#13
Open
muratsur wants to merge 8 commits into
Open
feat: add OpenCV inpaint fallback for videos where auto-detection fails#13muratsur wants to merge 8 commits into
muratsur wants to merge 8 commits into
Conversation
…ion fails GeminiWatermarkTool NCC-based detection occasionally fails when background content in the watermark region scores below the detection threshold. This script provides a fallback path using OpenCV inpainting. - Auto-detects the Gemini diamond by scanning the bottom-right corner - Builds a per-frame Otsu-thresholded + dilated mask for the diamond shape - Inpaints each frame with OpenCV TELEA algorithm preserving surrounding context - Re-encodes with ffmpeg keeping original audio - General CLI: python inpaint_fallback.py input.mp4 [output.mp4] Requires: opencv-python, ffmpeg on PATH Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend left padding from 10px to 18px and top padding to 15px so the diamond tip is fully covered in all observed 1080p portrait clips - Replace arrow Unicode character with ASCII to avoid cp1252 encode error on Windows terminals
gwt_snap_video.py: - Uses GeminiWatermarkTool image mode with --snap --fallback-region on each extracted frame, achieving the same reverse alpha blending quality as auto-detection mode for videos where video-mode NCC gate fails - Multi-scale NCC search (scales 16-160px) finds the watermark at the correct scale and position, then applies the calibrated alpha map precisely gwt_snap_hybrid.py: - Combines GWT snap reverse alpha blending with a residual cleanup pass - After GWT processes each frame, detects pixels still brighter than the expected background by more than 35 brightness units (tuned to avoid false positives on natural texture variation) - Inpaints only confirmed residual pixels with radius=3 TELEA for a clean finish - Handles both bright-background (wood texture) and dark-background frames Also fixes inpaint_fallback.py: - Default inpaint radius reduced to 3 (was 12) for better texture preservation - Asymmetric padding: 18px left, 15px top to fully cover diamond tip
Replace per-frame Otsu thresholding in the residual cleanup step with a fixed diamond mask computed once from the darkest-background reference frame. This prevents the inpainter from mistakenly treating skin tones and fabric textures as watermark pixels when people appear in the watermark region, eliminating the spurious artifact that was visible throughout the video (e.g. a white mark appearing over a character's chest/heart area). New approach: - build_fixed_mask(): scans 30 evenly-spaced frames, picks the one with the lowest Q25 background brightness, runs Otsu + dilation there once - inpaint_residual(): after GWT snap, checks only pixels that are BOTH inside the fixed mask AND still above bg+25; inpaints with the fixed mask (not re-thresholded per frame) - sample_background(): ring-samples the area around the watermark region to get a per-frame background brightness estimate Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…npaint Two key fixes: 1. Detection window widened (w-220 instead of w-200): The previous roi scan started at x=w-200=1720 for 1920px-wide video, but GWT reports the diamond at x=1704 — 16px outside the window. This left the diamond's left edge unmasked and visible after processing. Widening to w-220 ensures the full diamond is captured. 2. Dark-shadow detection with inpaint fallback (radius=15): GWT snap correctly finds the watermark (NCC=0.945) but over-subtracts the alpha on textured backgrounds (wood grain, burlap fabric), leaving a dark smudge in the diamond shape instead of a bright residual. Analysis: >30% of fixed-mask pixels darkened by >20 units on every frame. Fix: after GWT snap, check if >30% of mask pixels got significantly darkened; if so, revert to the original frame and inpaint with the fixed diamond mask at radius=15. Larger radius allows TELEA to reconstruct complex textures (directional fabric grain) from further context. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the complex shadow-restore/residual logic with a clean two-step approach: GWT snap without --denoise ai (avoids the blurry-square patch it caused on textured backgrounds), followed by TELEA inpaint at radius=12 over the full fixed diamond mask (covers the semi-transparent outline ring GWT leaves behind). Also increase Otsu dilation kernel to 7x7/iter=2 to ensure the outline pixels at the diamond edge are fully covered by the mask. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
|
Update: v12 — snap + TELEA inpaint, drop Root cause identified: GWT snap removes the diamond interior correctly via reverse alpha blending, but leaves a semi-transparent outline ring (~384 bright pixels) at the diamond edge. With Fix (now landed):
The radius=3 attempts in earlier versions were insufficient for the 72×72 diamond on textured burlap/fabric backgrounds. Radius=12 gives clean reconstruction as confirmed by single-frame comparison across r=3/7/12/20. Removed ~70 lines of the shadow-restore/residual-routing logic which was adding complexity without fixing the core outline artifact. |
All inpaint-based approaches (TELEA radius 3-30, full mask, outline-only) produced a visible texture smudge on dark burlap/fabric backgrounds because the fabric's diagonal weave cannot be reconstructed from local boundary info. New approach: exemplar-based patch copy using cv2.matchTemplate. - Build a boundary mask (outer ring of the diamond, clean fabric pixels). - Use masked TM_SQDIFF template matching to find the best-fitting fabric patch in a 300×240 px search window to the left of the watermark. - Copy that patch over the diamond area. - Feather only OUTSIDE the watermark mask (1→0 over 12 px) so no bright watermark pixels bleed back through the blend. - Falls back to TELEA r=7 if no suitable search region exists. SSD dropped 10× vs. the previous patch-copy attempt (fine step=1 vs step=4), giving a seamless result on the dark burlap frames at seconds 4-5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
|
Update: v14 — patch-copy + feathered blend, no GWT, no TELEA inpaint Root cause of the persistent smudge at seconds 4-5: the burlap/fabric diagonal weave pattern cannot be reconstructed by any local inpainting algorithm (TELEA radius 3–30 all gave the same visible patch). GWT snap over-subtracted on dark backgrounds, and TELEA fills center pixels from imperfect intermediate reconstructions. Fix: exemplar-based patch copy using
Best-match SSD dropped 10× vs. the previous attempt (fine search step=1, masked boundary matching). Single-frame comparison on the second-5 frame showed clean result with no visible diamond outline or smudge. |
…v18) Several bugs in the patch-copy approach caused wrong patches to be copied: 1. Search-area overlap: sx2 was set to wm_x1-2, allowing source patches starting at x=1702 that overlap the watermark at x=1704. Fixed to sx2 = wm_x1 - dw - 10 so the entire source patch clears the watermark. 2. Wrong search direction: searching leftward puts the person's arm/hand into the candidate area at later timestamps (~5.5s). Changed primary search to ABOVE the watermark (same x column, y-100 to y-15) which stays on the shirt fabric and avoids body parts. 3. Loose color validation: brightness-only check (±25) passed skin-toned patches that had similar mean brightness. Added per-channel BGR color check (±15) against the sampled background color to reject any patch whose median color diverges from the fabric. 4. Vertical range too large: 250 px above reached the person's neck and bead decorations. Clamped to 100 px to stay on the shirt. Also tightened the watermark mask: threshold lowered from bg+20 to bg+12 with larger dilation (5×5 ellipse, 2 iterations) to fully cover the semi-transparent edge gradient so no watermark pixels bleed through the feathered blend boundary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Author
|
Update: v15–v18 — patch-copy bug fixes Three bugs were causing wrong/skin-toned patches to be copied over the watermark:
Also tightened the watermark mask: threshold Grid comparison across t=4.0, 4.5, 5.0 shows clean fabric replacement with no diamond outline, no blob, no smudge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
inpaint_fallback.py— a standalone Python CLI script for removing the Gemini 3.5 diamond watermark from videos whereGeminiWatermarkTool-Videoauto-detection fails (NCC gate not met)python inpaint_fallback.py input.mp4 [output.mp4]When to use
Run this after
GeminiWatermarkTool-Videooutputs a[SKIP]message — e.g. when foreground content in the watermark region on all probe frames pushes NCC below the detection threshold.Requirements
pip install opencv-pythonffmpegon PATHTest plan
GeminiWatermarkTool-VideoSKIPs — confirm watermark is removed🤖 Generated with Claude Code