next-frame-lora

LoRA fine-tuning for temporally consistent video editing using Flux Klein 4B.

Fixes the flickering problem: AI models editing frames independently don't know what they did to previous frames, causing inconsistency between frames.

Results

Comparison (Original | Naive | LoRA):

The naive approach edits each frame independently, causing flickering. The LoRA approach uses temporal conditioning (referencing frames 5-20 steps back) to maintain consistency.

Method

Trains on triplets: (edited[t-k], original[t]) → edited[t]

• k randomly sampled from [5, 20] per frame during training
• Autoregressive inference: each output becomes reference for future frames
• Uses fal.ai API for editing and training

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
export FAL_KEY=your_key

Requirements: Python 3.10+, FFmpeg, FAL.ai API key

Pipeline

1. Data prep (extract frames → edit → create pairs)

python3 pipeline.py all --k-min 5 --k-max 20

Output: data/pairs.json with ~3k training pairs

2. Train (20-40 min on fal.ai)

python3 train.py --steps 1000

Output: LoRA URL saved to data/lora_url.txt

3. Evaluate

python3 eval.py baseline --video data/test/video.mp4
python3 eval.py lora --video data/test/video.mp4 --k-min 5 --k-max 20
python3 eval.py compare --video data/test/video.mp4 --with-lora

Output: data/test/video/output/comparison_with_lora.mp4

Files

data/
├── videos/          # input: training videos
├── frames/          # extracted at 8fps
├── edited/          # flux edits
├── pairs.json       # training manifest
├── lora_url.txt     # trained model
└── test/
    └── <video>/
        ├── frames/
        ├── naive/   # frame-by-frame (no consistency)
        ├── lora/    # with temporal conditioning
        └── output/  # comparison videos

Details

Training format:

• ROOT_start.jpg = original[t]
• ROOT_start2.jpg = edited[t-k] (reference)
• ROOT_end.jpg = edited[t] (target)
• ROOT.txt = caption

Inference:

• t < k_max: use naive edits as reference
• t ≥ k_max: use own output from t-k as reference (k random from [5,20])

Custom prompts: Edit VIDEO_PROMPTS dict in pipeline.py