Dear Authors,
First of all, thank you for your impressive work on this 6-DoF spatial reasoning benchmark. It provides very valuable insights for our ongoing research in Embodied AI and Vision-Language Models.
While I was studying the results in Table 6 to better understand the performance of various baselines, I encountered a small point of confusion regarding the Total scores. I was hoping you could help me clarify this to ensure I am interpreting the benchmark correctly.
Based on the dataset description, I noted the benchmark consists of 223 samples, distributed as 53/74/48/48 across the four categories. In my attempt to replicate the weighted average for the Total column using these counts, I noticed my calculated values slightly differ from the reported ones for a few models:
$$
\text{Total}_{\text{calc}} = \frac{(\text{Score}_{\text{pos-rel}} \times 53) + (\text{Score}_{\text{pos-abs}} \times 74) + (\text{Score}_{\text{ori-rel}} \times 48) + (\text{Score}_{\text{ori-abs}} \times 48)}{223}
$$
| Model |
Reported Sub-scores (rel. p / abs. p / rel. o / abs. o) |
Reported Total |
Calculated Total (Weighted) |
| GPT-4o-mini |
33.3 / 26.9 / 32.5 / 23.8 |
31.0 |
29.0 |
| SpaceLLaVA |
32.4 / 30.5 / 30.9 / 24.9 |
28.2 |
29.8 |
I was wondering if I might have misunderstood the weighting mechanism used for the final aggregation, or if there were specific subsets used for these models?
Could you please provide some guidance on how to correctly derive these total scores? We want to make sure our own comparisons are aligned with your evaluation methodology.
Thank you very much for your time and for this great contribution to the community.
Dear Authors,
First of all, thank you for your impressive work on this 6-DoF spatial reasoning benchmark. It provides very valuable insights for our ongoing research in Embodied AI and Vision-Language Models.
While I was studying the results in Table 6 to better understand the performance of various baselines, I encountered a small point of confusion regarding the Total scores. I was hoping you could help me clarify this to ensure I am interpreting the benchmark correctly.
Based on the dataset description, I noted the benchmark consists of 223 samples, distributed as 53/74/48/48 across the four categories. In my attempt to replicate the weighted average for the Total column using these counts, I noticed my calculated values slightly differ from the reported ones for a few models:
I was wondering if I might have misunderstood the weighting mechanism used for the final aggregation, or if there were specific subsets used for these models?
Could you please provide some guidance on how to correctly derive these total scores? We want to make sure our own comparisons are aligned with your evaluation methodology.
Thank you very much for your time and for this great contribution to the community.