Some questions seem to be too hard to calculate

[prnake]

First, thank you for commiting such a great IMO level evaluation dataset to the community. Each question is worth at least $200 on the market.

Problems created through crowdsourcing often suffer from numerical anomalies. In order to construct questions that are difficult enough to challenge large models, some contributors deliberately choose particularly cumbersome numbers. As a result, among the 50 published problems, several have overly complex numerical expressions in both the problem statements and the answers. Such numbers are too difficult to calculate for both humans and large models. Meanwhile, problems in the IMO tend to be relatively elegant.

[an1006634493]

Thank you for pointing out this issue! During our data collection process, we did encounter many problems featuring unusually complex numerical expressions. To address this, we have implemented the following strategy: during the difficulty review stage, if all tested LLMs fail solely due to numerical computation, we classify the problem as insufficiently challenging. This strategy has helped us filter out most problems whose difficulty stems solely from numerical complexity (more than 10 problems). However, it is true that some of the remaining problems do derive part of their difficulty from complex numerical expressions.
To help users better analyze the reasons for model errors, we will mark the four problems containing complex numerical expressions (Problem 12, 13, 15 and 21) in README.
Thank you again for your interest in our benchmark!