Hello,
Really apprecaite you guys open source this amazing work. I am trying to reproduce the results on infovqa dataset but get a about 80.6% accuracy. I am currently using the following setting up, Could you guys help me figuring out the correct setting up?
Reproduction Settings for InfoVQA
Model: TIGER-Lab/PixelReasoner-RL-v1 (HuggingFace)
Eval Data: JasperHaozhe/InfoVQA-EvalData-PixelReasoner (2801 samples)
Decoding
temperature: 0.0 (forced greedy in code when is_eval=True)
top_p: 1.0
top_k: -1
Image Resolution
MIN_PIXELS: 401408
MAX_PIXELS: 4014080
preprocessor_config.json min_pixels: changed from 3136 → 401408
Generation
num_episodes: 3
generate_max_len: 8192
system_prompt: vcot
vLLM
vllm: 0.7.3
gpu_memory_util: 0.85
tensor_parallel: 1
n_samples_per_prompt: 1
enable_prefix_caching: true
Environment
torch: 2.5.1
transformers: 4.49.0.dev0 (commit 9985d06)
ray: 2.42.0
GPU: 2× A100 80GB PCIe
MIN/MAX_PIXELS propagated via Ray RUNTIME_ENV_JSON
Result: 80.5% (2255/2801) — paper reports 84%.
Note: Same setup reproduces VStar correctly (86.1% vs paper 84%).
Hello,
Really apprecaite you guys open source this amazing work. I am trying to reproduce the results on infovqa dataset but get a about 80.6% accuracy. I am currently using the following setting up, Could you guys help me figuring out the correct setting up?
Reproduction Settings for InfoVQA
Model: TIGER-Lab/PixelReasoner-RL-v1 (HuggingFace)
Eval Data: JasperHaozhe/InfoVQA-EvalData-PixelReasoner (2801 samples)
Decoding
temperature: 0.0 (forced greedy in code when is_eval=True)
top_p: 1.0
top_k: -1
Image Resolution
MIN_PIXELS: 401408
MAX_PIXELS: 4014080
preprocessor_config.json min_pixels: changed from 3136 → 401408
Generation
num_episodes: 3
generate_max_len: 8192
system_prompt: vcot
vLLM
vllm: 0.7.3
gpu_memory_util: 0.85
tensor_parallel: 1
n_samples_per_prompt: 1
enable_prefix_caching: true
Environment
torch: 2.5.1
transformers: 4.49.0.dev0 (commit 9985d06)
ray: 2.42.0
GPU: 2× A100 80GB PCIe
MIN/MAX_PIXELS propagated via Ray RUNTIME_ENV_JSON
Result: 80.5% (2255/2801) — paper reports 84%.
Note: Same setup reproduces VStar correctly (86.1% vs paper 84%).