The inpainting effect on the first frame works fine, but from the first frame to the video, the inpainting effect appears very pool.

[JayChen7777]

Thnk you for your work! I have a small problem

109388395065_fps8.mp4

Here is my the first inpainting frame and groud_truth masked image.

Here is my cmd:
`
export CUDA_VISIBLE_DEVICES=0

model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V"
num_inference_steps=50
guidance_scale=6.0
num_videos_per_prompt=1
dtype="bfloat16"

inpainting_branches=(
/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch
)

id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints

lora_rank=256

inpainting_sample_ids=($(seq 0 0))

inpainting_frames=49
down_sample_fps=8
overlap_frames=0
image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video"

data_kind="test"
if [ "$data_kind" == "test" ]; then
inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv"
elif [ "$data_kind" == "val" ]; then
inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv"
elif [ "$data_kind" == "train" ]; then
inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv"
else
echo "data_kind must be test or train or val"
exit 1
fi

prev_clip_weight=0.0

img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev"

llm_model="gpt-4o"

dilate_size=0

output_base_path="./inp_fps${down_sample_fps}dilate${dilate_size}_${data_kind}_no_prompt_1003"

if [ "$llm_model" != "None" ]; then
output_base_path="${output_base_path}_${llm_model}"
fi

if [ "$overlap_frames" != "0" ]; then
output_base_path="${output_base_path}overlap${overlap_frames}"
fi

if [ ! -d "$output_base_path" ]; then
mkdir -p "$output_base_path"
chmod -R 777 "$output_base_path"
fi

for inpainting_branch in "${inpainting_branches[@]}"; do
for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do
output_path="${output_base_path}/${inpainting_sample_id}.mp4"
python inpaint.py
--prompt "$prompt"
--model_path "$model_path"
--inpainting_branch "$inpainting_branch"
--output_path "$output_path"
--num_inference_steps "$num_inference_steps"
--guidance_scale "$guidance_scale"
--num_videos_per_prompt "$num_videos_per_prompt"
--dtype "$dtype"
--generate_type "i2v_inpainting"
--inpainting_mask_meta "$inpainting_mask_meta"
--inpainting_sample_id "$inpainting_sample_id"
--inpainting_frames "$inpainting_frames"
--image_or_video_path "$image_or_video_path"
--first_frame_gt
--replace_gt
--mask_add
--down_sample_fps $down_sample_fps
--overlap_frames $overlap_frames
--prev_clip_weight $prev_clip_weight
--img_inpainting_model $img_inpainting_model
--llm_model $llm_model
--dilate_size $dilate_size
done
done`

[yxbian23]

Thnk you for your work! I have a small problem

109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.

Here is my cmd: ` export CUDA_VISIBLE_DEVICES=0

model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16"

inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch )

id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints

lora_rank=256

inpainting_sample_ids=($(seq 0 0))

inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video"

data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi

prev_clip_weight=0.0

img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev"

llm_model="gpt-4o"

dilate_size=0

output_base_path="./inp_fps${down_sample_fps}dilate${dilate_size}_${data_kind}_no_prompt_1003"

if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi

if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}overlap${overlap_frames}" fi

if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi

for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done`

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

[JayChen7777]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

[JayChen7777]

Also, I want to know the amount of data required for fine-tuning such masks.

[yxbian23]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊
You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.

2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

[yxbian23]

Also, I want to know the amount of data required for fine-tuning such masks.

I wish I can tell you a specific number😭 But since I haven't done such finetuning before, I can only infer by our experience on training the VideoPainter. I think around 1000 additional training data maybe enough for the model to adapt (Definitely more is better)

[JayChen7777]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊 You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.
2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

Thank you for your detailed reply! I retrained the model using square masks (without using your released weights). On 4*80G A100 GPUs with a per-GPU batch size of 1 and checkpoint 1024, I found that the model already shows certain inpainting capability on square masks.
I believe that "The main performance drop is likely caused by the square mask itself rather than the background change." is correct. Next, I will try more complex scenarios.

However, I’ve noticed a new issue: after inpainting, the video sometimes shows color changes. Is this a normal phenomenon?

175976484536_fps8.1.mp4

validation_00001_1024_video_0_False_A_man_dances_joyfully_on_.mp4

[yxbian23]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊 You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.
2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

Thank you for your detailed reply! I retrained the model using square masks (without using your released weights). On 4*80G A100 GPUs with a per-GPU batch size of 1 and checkpoint 1024, I found that the model already shows certain inpainting capability on square masks. I believe that "The main performance drop is likely caused by the square mask itself rather than the background change." is correct. Next, I will try more complex scenarios.

However, I’ve noticed a new issue: after inpainting, the video sometimes shows color changes. Is this a normal phenomenon?

175976484536_fps8.1.mp4
validation_00001_1024_video_0_False_A_man_dances_joyfully_on_.mp4

Happy to see you made it work!

The first video result is normal. The little color difference usually appear when the model hasn't fully converge.

The second video result seems a little bit strange, but I think maybe it is also because the incomplete converge (because it can currently still inpaint the region).

[JayChen7777]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊 You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.
2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

Thank you for your detailed reply! I retrained the model using square masks (without using your released weights). On 4*80G A100 GPUs with a per-GPU batch size of 1 and checkpoint 1024, I found that the model already shows certain inpainting capability on square masks. I believe that "The main performance drop is likely caused by the square mask itself rather than the background change." is correct. Next, I will try more complex scenarios.
However, I’ve noticed a new issue: after inpainting, the video sometimes shows color changes. Is this a normal phenomenon?
175976484536_fps8.1.mp4
validation_00001_1024_video_0_False_A_man_dances_joyfully_on_.mp4

Happy to see you made it work!

The first video result is normal. The little color difference usually appear when the model hasn't fully converge.

The second video result seems a little bit strange, but I think maybe it is also because the incomplete converge (because it can currently still inpaint the region).

Thanks! I understand and I will provide the results after further convergence.

[JayChen7777]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊 You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.
2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

Thank you for your detailed reply! I retrained the model using square masks (without using your released weights). On 4*80G A100 GPUs with a per-GPU batch size of 1 and checkpoint 1024, I found that the model already shows certain inpainting capability on square masks. I believe that "The main performance drop is likely caused by the square mask itself rather than the background change." is correct. Next, I will try more complex scenarios.
However, I’ve noticed a new issue: after inpainting, the video sometimes shows color changes. Is this a normal phenomenon?
175976484536_fps8.1.mp4
validation_00001_1024_video_0_False_A_man_dances_joyfully_on_.mp4

Happy to see you made it work!

The first video result is normal. The little color difference usually appear when the model hasn't fully converge.

The second video result seems a little bit strange, but I think maybe it is also because the incomplete converge (because it can currently still inpaint the region).

I found that when I set replace_gt == True and dilate_size = 32, the color change problem is solved.
However, I also found that this effect cannot generalize to videos with frequent mask changes.
For example, when removing subtitles from a video, the text may vary in length and frequently change position, which makes the results unstable.

109388395065_fps8.mp4

[yxbian23]

Thnk you for your work! I have a small problem
109388395065_fps8.mp4
Here is my the first inpainting frame and groud_truth masked image.
Here is my cmd: export CUDA_VISIBLE_DEVICES=0 model_path="/mmu_vcg2_wjc_ssd/xxx/models/zai-org/CogVideoX-5b-I2V" num_inference_steps=50 guidance_scale=6.0 num_videos_per_prompt=1 dtype="bfloat16" inpainting_branches=( /mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainter/checkpoints/branch ) id_adapter_resample_learnable_path=/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VideoPainter/VideoPainterID/checkpoints lora_rank=256 inpainting_sample_ids=($(seq 0 0)) inpainting_frames=49 down_sample_fps=8 overlap_frames=0 image_or_video_path="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/videovo/raw_video" data_kind="test" if [ "$data_kind" == "test" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_test_dataset.csv" elif [ "$data_kind" == "val" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_val_dataset.csv" elif [ "$data_kind" == "train" ]; then inpainting_mask_meta="/mmu_vcg2_wjc_ssd/xxx/models/TencentARC/VPBench/pexels_videovo_train_dataset.csv" else echo "data_kind must be test or train or val" exit 1 fi prev_clip_weight=0.0 img_inpainting_model="/mmu_vcg2_wjc_ssd/xxx/models/black-forest-labs/FLUX.1-Fill-dev" llm_model="gpt-4o" dilate_size=0 output_base_path="./inp_fps${down_sample_fps}_dilate_${dilate_size}_${data_kind}_no_prompt_1003" if [ "$llm_model" != "None" ]; then output_base_path="${output_base_path}_${llm_model}" fi if [ "$overlap_frames" != "0" ]; then output_base_path="${output_base_path}_overlap_${overlap_frames}" fi if [ ! -d "$output_base_path" ]; then mkdir -p "$output_base_path" chmod -R 777 "$output_base_path" fi for inpainting_branch in "${inpainting_branches[@]}"; do for inpainting_sample_id in "${inpainting_sample_ids[@]}"; do output_path="${output_base_path}/${inpainting_sample_id}.mp4" python inpaint.py --prompt "$prompt" --model_path "$model_path" --inpainting_branch "$inpainting_branch" --output_path "$output_path" --num_inference_steps "$num_inference_steps" --guidance_scale "$guidance_scale" --num_videos_per_prompt "$num_videos_per_prompt" --dtype "$dtype" --generate_type "i2v_inpainting" --inpainting_mask_meta "$inpainting_mask_meta" --inpainting_sample_id "$inpainting_sample_id" --inpainting_frames "$inpainting_frames" --image_or_video_path "$image_or_video_path" --first_frame_gt --replace_gt --mask_add --down_sample_fps $down_sample_fps --overlap_frames $overlap_frames --prev_clip_weight $prev_clip_weight --img_inpainting_model $img_inpainting_model --llm_model $llm_model --dilate_size $dilate_size done done

Hi, this is likely because VideoPainter haven't seen such squre and static masks during training. Similar issue can be found in #38. I recommend to conduct a finetuning of such masks to have better performance~

Thank you for your reply! I also want to know whether this method is only suitable for relatively static scenes where the background bounding box does not change significantly. When the video is more complex and the background changes frequently, the performance seems to degrade a lot. I'm not sure whether this is caused by the square mask or by the background switching, since the examples you provided all show slowly changing masks with only a single object being inpainted throughout the video.

Thanks for the great question! 😊 You’re right to notice the difference in performance across scene complexity — here are a few clarifications that might help:

1. The main performance drop is likely caused by the square mask, rather than background change.
   Our model was trained entirely with segmentation-based masks (object-level masks), not arbitrary square / bounding-box regions.
   This is why the examples we show focus on inpainting a specific object rather than inpainting a free-form region.
   When a square region is used, the model simply hasn’t seen this pattern during training, so the results can degrade.
2. Background changes should generally be less of an issue.
   The training data comes from a wide variety of real-world Internet videos, covering diverse scenes, camera motions, and environments. The dataset is fully open-sourced here: https://huggingface.co/datasets/TencentARC/VPData
   So the model should hopefully have already learned to be relatively robust to background variation.

Thank you for your detailed reply! I retrained the model using square masks (without using your released weights). On 4*80G A100 GPUs with a per-GPU batch size of 1 and checkpoint 1024, I found that the model already shows certain inpainting capability on square masks. I believe that "The main performance drop is likely caused by the square mask itself rather than the background change." is correct. Next, I will try more complex scenarios.
However, I’ve noticed a new issue: after inpainting, the video sometimes shows color changes. Is this a normal phenomenon?
175976484536_fps8.1.mp4
validation_00001_1024_video_0_False_A_man_dances_joyfully_on_.mp4

Happy to see you made it work!
The first video result is normal. The little color difference usually appear when the model hasn't fully converge.
The second video result seems a little bit strange, but I think maybe it is also because the incomplete converge (because it can currently still inpaint the region).

Thanks! I understand and I will provide the results after further convergence.

Hi @JayChen7777 , I think this is within the expectations~

The core reason is that the original VideoPianter is only trained with a normal natural video and a continuous mask sequence of a specific objects in the video.

The case you show here belongs to another inpainting scenario, where only the texts rendering region is inpainted and the background has a lot changes. Actually, since the original VideoPianter is trained to handle the most general case and at that time we have limited computing resources, we even filtered all of this text rendering cases and the cases with sudden background changes~ Thar's why you will see low quality results on these Ad videos on Douyin or Tiktok~

So if you are target on the inpainting tasks you showed ("change text rendering with many video background changes"), it would be better to have more data of this type to finetune (more data is better, and I think the structure will need no modifications)~

[JayChen7777]

Hi ~ Thanks! I still have three questions:

1. I found that GPU memory usage is quite high — about 48GB. Processing a 49-frame video takes around 3:46 (226 seconds) on a single 80GB A800. For long videos, we need to process them in chunks, which actually depends on shot segmentation. In that case, the background usually doesn’t change much. I tried the torchao quantization from CogVideoX-5b-I2V, and the memory usage dropped from ~48GB → ~36GB. However, the time cost is too high. Are there any methods to speed this up?

2. I noticed that the training takes quite a long time, so I want to know the recommended number of epochs for convergence.

3. During inference, using --replace_gt helps reduce color changes, since it only replaces the masked regions. However, with this setting, there may be a visible line along the edges. How can this be resolved?

177799690406_part00006_00055.mp4