Skip to content

README.md

Latest commit

 

History

History
85 lines (56 loc) · 3.48 KB

README.md

File metadata and controls

85 lines (56 loc) · 3.48 KB
Raw

Sim2Real Retexturing

Two-stage FLUX.1-dev LoRA pipeline that transforms synthetic GarmentCode renders into photorealistic try-on images.

  • Stage 1 (f_texture): normal map + segmentation + caption → photorealistic try-on image
  • Stage 2 (f_paired): normal map + identity map + caption → paired person image I_p (same person, different upper garment)

Both stages use the same LoRA checkpoint fine-tuned on the FIT dataset.

Installation

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

Model Weights

  • LoRA checkpoint — download default.safetensors from Yuanhao-Harry-Wang/FIT-sim2real and place it under checkpoint/.
  • Base model (black-forest-labs/FLUX.1-dev) — downloaded automatically, or set --flux_model_path.
  • Quality check model (briaai/RMBG-2.0) — downloaded automatically when --use_quality_check is set.

Caption Format

Captions must follow a 3-sentence structure separated by ". ":

  1. Person description — "A young man stands against a soft studio background."
  2. Upper garment — "He wears a floral short-sleeved shirt with visible shoulder seams."
  3. Lower garment — "His relaxed-fit trousers display a checkered print."

Stage 2 prompt construction: sentences are recombined across the target and identity captions to describe the identity person wearing the target's upper garment:

prompt = [identity sentence 1] + [target sentence 2] + [identity sentence 3]

Quality Check

--use_quality_check filters outputs via silhouette IoU (RMBG-2.0 vs. input normal map mask). This catches rare failures — background hallucination or missing limbs — that can occur but are uncommon.

Quick Start

python run_example.py --pair 1                              # pair 1a / 1b
python run_example.py --pair 2 --use_quality_check         # pair 2a / 2b, with quality check

Outputs saved to examples/output/pair{N}/: stage1_target.png, stage1_identity.png, identity_map.png, stage2_paired.png, grid.png.

Inputs

Input Description
normal/ Person normal map, 768×1024 RGB
segmentation/ Body segmentation map (grayscale), 768×1024. Values: 0 = background, 85 = upper garment, 170 = lower garment, 255 = body / hair / other
hairmask/ Binary hair mask — preserves hair when constructing the identity map (only required at stage 2)
caption/ Text caption in the 3-sentence format above

Note: The segmentation map is not described in the paper. It was added in practice as a third model condition to resolve regions that are ambiguous from the normal map alone — particularly the upper/lower garment boundary, where similar surface orientations make garment-type disambiguation difficult.

Arguments

Argument Default Description
--pair 1 Example pair to process (1 = 1a/1b, 2 = 2a/2b)
--flux_model_path black-forest-labs/FLUX.1-dev HuggingFace ID or local path to FLUX.1-dev
--checkpoint_path checkpoint/ Directory containing default.safetensors
--output_dir examples/output Directory to write outputs
--use_quality_check off Enable silhouette IoU quality filtering
--rmbg_model_path briaai/RMBG-2.0 HuggingFace ID or local path to RMBG-2.0
--threshold 0.93 IoU threshold for quality check