Skip to content

Commit 52d7a1e

Browse files
committed
[Fix] Disable use_cache during export dummy forward for custom configs
The export step's collect_shared_input_modules() runs a dummy forward pass to trace shared input modules for resmoothing/requantization. This forward call hits the same AttributeError as PR #1324 on configs that don't assign use_cache (e.g., stepfun-ai/Step-3.5-Flash's Step3p5Config): AttributeError: 'Step3p5Config' object has no attribute 'use_cache' Wrap the dummy forward with the _disable_use_cache context manager so it sets/restores config.use_cache around the call, mirroring the calibration fix from #1324. Covers all callsites since both unified_export_hf and plugins/vllm_fakequant_hf funnel through collect_shared_input_modules. Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
1 parent 097293b commit 52d7a1e

1 file changed

Lines changed: 5 additions & 1 deletion

File tree

modelopt/torch/export/unified_export_hf.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@
5555
from modelopt.torch.quantization.nn import SequentialQuantizer, TensorQuantizer
5656
from modelopt.torch.quantization.qtensor import MXFP8QTensor, NVFP4QTensor
5757
from modelopt.torch.quantization.utils import fsdp2_aware_weight_update, quantizer_attr_names
58+
from modelopt.torch.utils.dataset_utils import _disable_use_cache
5859

5960
try:
6061
from modelopt.torch.sparsity.attention_sparsity.conversion import export_sparse_attention_config
@@ -213,11 +214,14 @@ def _output_hook(module, input, output):
213214
if not handles:
214215
return input_to_linear, output_to_layernorm
215216

216-
# Run dummy forward pass to collect modules sharing same input
217+
# Run dummy forward pass to collect modules sharing same input.
218+
# `_disable_use_cache` keeps the probe forward working on configs that don't
219+
# set `use_cache` (e.g., stepfun-ai/Step-3.5-Flash's Step3p5Config).
217220
try:
218221
with (
219222
torch.no_grad(),
220223
set_quantizer_by_cfg_context(model, [{"quantizer_name": "*", "enable": False}]),
224+
_disable_use_cache(model),
221225
):
222226
dummy_forward_fn()
223227
finally:

0 commit comments

Comments
 (0)