Skip to content

best hyperparameter setting for infovqa #41

Description

@LeiGong0125Carrot

Hello,

Really apprecaite you guys open source this amazing work. I am trying to reproduce the results on infovqa dataset but get a about 80.6% accuracy. I am currently using the following setting up, Could you guys help me figuring out the correct setting up?

Reproduction Settings for InfoVQA

Model: TIGER-Lab/PixelReasoner-RL-v1 (HuggingFace)
Eval Data: JasperHaozhe/InfoVQA-EvalData-PixelReasoner (2801 samples)

Decoding

temperature: 0.0 (forced greedy in code when is_eval=True)
top_p: 1.0
top_k: -1

Image Resolution

MIN_PIXELS: 401408
MAX_PIXELS: 4014080
preprocessor_config.json min_pixels: changed from 3136 → 401408

Generation

num_episodes: 3
generate_max_len: 8192
system_prompt: vcot

vLLM

vllm: 0.7.3
gpu_memory_util: 0.85
tensor_parallel: 1
n_samples_per_prompt: 1
enable_prefix_caching: true

Environment

torch: 2.5.1
transformers: 4.49.0.dev0 (commit 9985d06)
ray: 2.42.0
GPU: 2× A100 80GB PCIe

MIN/MAX_PIXELS propagated via Ray RUNTIME_ENV_JSON

Result: 80.5% (2255/2801) — paper reports 84%.
Note: Same setup reproduces VStar correctly (86.1% vs paper 84%).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions