OpenProblemBench：针对基础理论科学中未解决问题的AI基准测试
OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences
浏览论文内容
中文总结 AI 辅助
OpenProblemBench是含82个数理理论未解决问题的AI基准，经评估GPT-6-Astra解决率最高达14.0%，可用于研究AI在基础理论科学领域的能力与局限。
中文摘要 AI 辅助
通用人工智能的下一个前沿领域是解决未解决的科学问题，这需要能够评估超越现有知识进展的基准测试。我们推出OpenProblemBench，这是一个包含82个未解决问题的基准测试，这些问题来自数学和理论物理学文献。每个问题都提供了研究背景、假设以及调查该问题所需的前期进展。我们选择的问题其拟议解决方案能够对其决定性的数学或计算主张进行相对清晰的验证。四个评估模型在不参考解决方案的情况下，独立评估每个提交内容的正确性、完整性和进展程度。在七个评估配置中，GPT-6-Astra的平均判定解决率最高，为14.0%，相比之下，评估的全尺寸开源模型为5.5-6.7%，Flash模型为2.4-3.7%。案例比较表明，更好的结果与问题表示的变化、超越有限证据的一般性论证以及完成解决方案所需步骤的证明相关。通过将评估建立在研究文献中出现的问题基础上，OpenProblemBench为研究AI作为基础理论科学贡献者的能力和局限性提供了一个场景。
英文摘要
The next frontier for artificial general intelligence is tackling unresolved scientific problems, calling for benchmarks that assess progress beyond established knowledge. We introduce OpenProblemBench, a benchmark of 82 unresolved problems drawn from the mathematics and theoretical physics literature. Each problem supplies the research context, assumptions, and prior progress needed to investigate the question. We select problems whose proposed solutions admit comparatively clear checks of their decisive mathematical or computational claims. Four evaluator models independently assess the correctness, completeness, and degree of progress of each submission without reference solutions. Across seven evaluated configurations, GPT-6-Astra achieves the highest mean judged solve rate of 14.0%, compared with 5.5-6.7% for the evaluated full-size open models and 2.4-3.7% for Flash models. Case comparisons connect stronger outcomes to changes in problem representation, general arguments that extend beyond finite evidence, and proofs of the steps needed to complete a solution. By grounding evaluation in questions arising from the research literature, OpenProblemBench provides a setting for investigating the capabilities and limitations of AI as a contributor to foundational theoretical science.
发表机构
- University of Science and Technology of China（中国科学技术大学）
- Institute of Theoretical Physics, Chinese Academy of Sciences（中国科学院理论物理研究所）
- Hefei National Laboratory（合肥国家实验室）
- Hefei National Research Center for Physical Sciences at the Microscale（合肥微尺度物质科学国家研究中心）
- School of Physical Sciences, University of Science and Technology of China（中国科学技术大学物理学院）
- Institute for Advanced Algorithms Research（先进算法研究院）
- Endless Frontier（无尽前沿（Endless Frontier））
机构由 AI 辅助整理，请以论文原文为准。
