What is the truth about the latest AI mathematics capabilities? All models perform less than 5% o... скачать в хорошем качестве

What is the truth about the latest AI mathematics capabilities? All models perform less than 5% o...

[Compass for the AI ​​Era] Paper Commentary Series Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, Martin Vechev https://arxiv.org/abs/2503.21934 ⭐️Story Description The story of this video is about a fisherman grandfather teaching Nyanta about the relationship between AI and mathematics. The latest AI not only performs numerical calculations but also attempts proofs, but the evaluation at the Math Olympiad showed noticeable logical errors and lack of creativity, and it was found that there are still issues with proof capabilities. ⭐️Key points 1. Key findings: The study evaluated six state-of-the-art LLM reasoning ability models against six USAMO2025 problems, and all models performed significantly poorly in terms of mathematical rigor. The highest scores were less than 5% on average, and common failure patterns were identified, including logical fallacies, unjustified assumptions, and lack of creativity in proof generation. This indicates a serious gap in AI mathematical ability assessment. 2. Methodology: The study used six problems from USAMO2025 to evaluate the full mathematical proof ability of each model. Four math experts (former IMO team members) performed human expert scoring, and each problem was independently evaluated by two people. Failure modes were identified through detailed analysis of reasoning traces and classified into four categories: logical fallacies, assumptions, creativity, and algebra/arithmetic. Evaluation on a more diverse set of problems could be considered as an improvement. 3. Research limitations: This study focuses on only a single [Mathematics Olympiad] ([USAMO2025]), which limits the generalization of the results. In addition, [human expert grading] may not be able to completely eliminate subjectivity. [Automatic grading limitations] have also been shown, and standardization of grading is a challenge. To address these limitations, additional research is needed that uses a broader set of mathematical problems and clarifies the grading criteria. 4. Related work: The paper mentions existing [AI mathematics ability evaluation] benchmarks such as [MathArena], pointing out that these only evaluate the final numerical answer and ignore the rigorous [proof generation] and [AI mathematical reasoning] processes. It also describes the difference with evaluation approaches that use formal verification tools such as Lean. This study focuses on the evaluation of [mathematical proofs] in natural language, overcoming the limitations of existing research. 5. Future impact: This study clarifies the current limitations in [AI mathematical reasoning] and [proof generation] and provides important guidance for improving [LLM reasoning ability] in the future. Identifying logical errors and limitations in automatic scoring will lead to the development of more effective training methods. Comparisons between models such as CLAUDE3.7 and O3-MINI will provide a basis for understanding the strengths and weaknesses of various approaches and may establish new standards for AI mathematics ability assessment. ▶︎Members only! Early access to videos here:    / @compassinai   ▶︎Qiita: https://qiita.com/compassinai Arxiv monthly rankings now available!