Feat/qwen3.5 fp8toint8

[1235v]

FP8 → INT8 Quantization Pipeline

This PR introduces a data-free FP8-to-INT8 conversion pipeline. It automatically dequantizes block-wise FP8 weights to BF16, then applies W8A8 PTQ via llmcompressor, producing an INT8 checkpoint ready for end-to-end inference.

---

How It Works

• Dequantize FP8 weights (block-wise weight × scale) → temporary BF16 checkpoint
• Apply W8A8 PTQ via llmcompressor (model_free_ptq) on the BF16 model
• Save INT8 checkpoint, auto-cleanup temporary files

---

Validated Models

Model	Type	FP8 Granularity
Qwen3.5-27B-FP8	Dense	Block-wise (128×128)
Qwen3.5-35B-A3B-FP8	MoE	Block-wise (128×128)
Qwen3.6-27B-FP8	Dense	Block-wise (128×128)
Qwen3-30B-A3B-FP8	MoE	Block-wise (128×128)
GLM-4.5-Air-FP8	MoE	Block-wise (128×128)
Deepseek-coder-v2-lite-instruct-FP8	MoE	Per-tensor
Llama-3.1-8B-Instruct-FP8-dynamic	Dense	Per-channel

• 7 models, 3 FP8 granularities (block-wise / per-channel / per-tensor), full coverage of Dense, MoE, and multimodal architectures, all verified with end-to-end inference.

---

How to Use FP8TOINT8 Quantization

cd FlagScale

# Quantization command
python run.py --config-path examples/qwen35/conf/inference --config-name compress_fp8toint8 \
    experiment.envs.MODEL_ID=/path/to/fp8_model \
    experiment.envs.SAVE_DIR=/path/to/output

tail -f outputs/fp8toint8_example/compress_logs/host_0_localhost.output

# End-to-end inference command
CUDA_VISIBLE_DEVICES=4,5,6,7 HIP_VISIBLE_DEVICES=4,5,6,7 vllm serve \
    ./models \
    --tensor-parallel-size 4 \
    --max-model-len 262144 \
    --port 8001 \
    --no-enable-log-requests \
    --no-enable-prefix-caching

[cyber-pioneer]

Add usage doc

[cyber-pioneer]

remove chinese

[rdzhu225]

Please remove commented-out code. If you need to document why the approach changed, add a comment explaining the reason.

The merge-base changed after approval.