arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.04872cs.AI
收藏

FrontierMath: 一个评估人工智能高级数学推理能力的基准

FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

Epoch AI;University of Leicester(利兹大学);RWTH Aachen University(亚琛工业大学);King’s College London(伦敦大学国王学院);University of Bristol(布里斯托大学);Iowa State University(爱荷华州立大学);MIT(麻省理工学院);University of North Carolina(北卡罗来纳大学);CNRS - Université Paris-Saclay(法国国家科学研究中心-巴黎-萨克雷大学);University of Siegen(锡根大学);Knox College at Charlotte(夏洛特克恩学院);James Madison University(詹姆斯麦迪逊大学);ICMC, USP(巴西圣保罗大学ICMC分校);Masaryk University(马萨里克大学);UC Berkeley(伯克利加州大学);Cornell University(康奈尔大学);Rutgers University(罗格斯大学);Harvard University(哈佛大学);ETH Zurich(苏黎世联邦理工学院);Independent(独立研究者)
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira… 展开作者

Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, Mark Wildon

AI总结:

FrontierMath是一个由专家数学家设计的数学问题基准,用于评估AI在高级数学推理能力上的表现,揭示了当前AI与数学界能力之间的显著差距。

摘要:

我们介绍了FrontierMath,一个由专家数学家创作和审核的数百个原始、极具挑战性的数学问题的基准。这些问题涵盖了现代数学的主要分支——从数论和实分析中的计算密集型问题到代数几何和范畴论中的抽象问题。解决一个典型问题需要相关数学分支的研究人员花费数小时的努力,而对于最高难度的问题,则需要数天时间。FrontierMath使用新发表的、未发表的问题和自动化验证来可靠地评估模型,同时最小化数据污染的风险。当前最先进的AI模型只能解决不到2%的问题,揭示了人工智能能力与数学界实力之间的巨大差距。随着AI系统朝着专家级数学能力发展,FrontierMath提供了一个严谨的测试平台,量化了它们的进步。

英文摘要:

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑