Reproduction gap for MiniCPM-V-4.6 on AI2D_TEST and MMMU_DEV_VAL with VLMEvalKit

[github-haoyan]

Hi, I am trying to reproduce the MiniCPM-V-4.6 results reported in the README using VLMEvalKit, but I observe a noticeable gap.

Model:

• openbmb/MiniCPM-V-4.6

Evaluation tool:

• VLMEvalKit
• Datasets: AI2D_TEST, MMMU_DEV_VAL
• Judge: exact_matching
• Mode: all

Command:

python run.py \
  --data MMMU_DEV_VAL AI2D_TEST \
  --model MiniCPM-V-4_6 \
  --mode all \
  --judge exact_matching \
  --reuse \
  --verbose

Dataset | Observed
AI2D_TEST | 78.76
MMMU_DEV_VAL | 42.56

Dataset | README
AI2D | 84.2
MMMU | 53.6

I see this warning during generation: Setting pad_token_id to eos_token_id:248044 for open-end generation.

I am not using an official MiniCPM-V-4.6 adapter from upstream VLMEvalKit. I added a local MiniCPM_V_4_6 adapter under vlmeval/vlm/minicpm_v.py, registered as MiniCPM-V-4_6. It inherits MiniCPM_V_4 for prompt building / CoT / answer extraction, and only changes loading/generation to use AutoModelForImageTextToText and AutoProcessor for openbmb/MiniCPM-V-4.6.

Parameters:
downsample_mode = "16x"
max_slice_nums = 36
num_beams = 3
do_sample = False
max_new_tokens = 2048 for CoT datasets
max_new_tokens = 1024 for non-CoT datasets
add_generation_prompt = True
processor_kwargs.images_kwargs.downsample_mode = "16x"
processor_kwargs.images_kwargs.max_slice_nums = 36

Could you please help check whether my evaluation setup is correct?

If there is any mismatch with the official evaluation protocol, could you suggest the recommended way to reproduce the reported AI2D/MMMU results for MiniCPM-V-4.6?

Please let me know if additional logs, prediction files, or environment details would be useful.

[tc-mb]

Hi, thanks for the detailed report. The gap is likely caused by these issues:

1. downsample_mode must be passed to both processor and model.generate()

v4.6 requires downsample_mode to be passed to both apply_chat_template (for correct visual token count) and generate() (for the vision encoder). If it's only passed on the processor side, the vision encoder behavior will mismatch the expected token count, directly hurting accuracy. This would especially explain the large gap on MMMU where fine-grained visual understanding is critical.

Make sure your adapter does:

model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=...)

2. Try downsample_mode="4x" instead of "16x"

Both AI2D (scientific diagrams) and MMMU (college-level multimodal problems with charts, formulas, sheet music, etc.) are detail-heavy benchmarks. "4x" preserves 4x more visual tokens than "16x". The official reported scores were most likely evaluated with "4x".

3. Inheriting from MiniCPM_V_4 is not compatible with v4.6

v4.6 uses a Qwen3.5-based tokenizer and is loaded via native AutoModelForImageTextToText, while v4 is an entirely different architecture. The chat template format, CoT prompts, and answer extraction logic inherited from MiniCPM_V_4 won't match v4.6's actual output -- particularly problematic for MMMU's CoT answer extraction.

I'd recommend writing the adapter directly from the VLMEvalKit base class rather than deriving from MiniCPM_V_4.

---

Happy to help if gaps remain after fixing these.

[github-haoyan]

Thanks for the clarification.

For point 1, I checked the adapter used in my experiments. downsample_mode is already passed to both the processor and the model:

inputs = self.processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
    processor_kwargs={
        "images_kwargs": {
            "downsample_mode": self.downsample_mode,
            "max_slice_nums": self.max_slice_nums,
        }
    },
).to(self.model.device)

generated_ids = self.model.generate(
    **inputs,
    downsample_mode=self.downsample_mode,
    **default_kwargs,
)

Therefore, point 1 should already be covered.

For point 2, I am intentionally using downsample_mode="16x" because I am comparing against the reported 16x results.

For point 3, I agree that inheriting from MiniCPM_V_4 may introduce incompatible behavior. However, simply inheriting from BaseModel and removing the v4-specific prompt, CoT, resizing, and answer extraction logic does not define the correct MiniCPM-V-4.6 evaluation protocol.

I did not find native support for MiniCPM-V-4.6 in the official VLMEvalKit repository. If an official MiniCPM-V-4.6 VLMEvalKit adapter or evaluation script is available, could you please share the corresponding file, commit, or implementation details? I would prefer to align directly with the official evaluation pipeline rather than reproduce it through trial and error.