arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.11652cs.CL
收藏

ACADREASON:探索推理模型在学术研究问题上的极限

ACADREASON: Exploring the Limits of Reasoning Models with Academic Research Problems

OPPO AI Agent Team(OPPO 人工智能代理团队)
Xin Gui, King Zhu, JinCheng Ren, Qianben Chen, Zekun Moore Wang, Yizhi LI, Xinpeng Liu, Xiaowan Li, Wenli Ren, Linyu Miao, Tianrui Qin, Ziqi Shu, He Zhu, Xiangr… 展开作者

Xin Gui, King Zhu, JinCheng Ren, Qianben Chen, Zekun Moore Wang, Yizhi LI, Xinpeng Liu, Xiaowan Li, Wenli Ren, Linyu Miao, Tianrui Qin, Ziqi Shu, He Zhu, Xiangru Tang, Dingfeng Shi, Jiaheng Liu, Yuchen Eleanor Jiang, Minghao Liu, Ge Zhang, Wangchunshu Zhou

AI总结:

针对现有学术基准推理深度不足,提出Acadreason基准,含50个专家标注的跨五领域学术问题,评估显示主流LLM和智能体得分普遍偏低,揭示其在高水平学术推理上的能力差距。

摘要:

近年来,大型语言模型(LLM)和智能体的研究焦点已日益从展示新颖能力转向复杂推理和应对挑战性任务。然而,现有评估主要集中于数学/编程竞赛或一般任务,而现有的多领域学术基准缺乏足够的推理深度,使得该领域缺少一个用于高级推理的严格基准。为填补这一空白,我们引入了Acadreason基准,旨在评估LLM和智能体获取学术知识并基于其进行推理的能力。该基准包含50个由专家标注的学术问题,涵盖计算机科学、经济学、法学、数学和哲学五个高推理需求领域。所有问题均源自近年来的顶级出版物,并经过严格的标注和质量控制,以确保其既具有挑战性又可回答。我们对超过10个主流LLM和智能体进行了系统评估。结果表明,大多数LLM得分低于20分,即使是前沿的GPT-5也仅获得16分。虽然智能体取得了更高的分数,但无一超过40分。这展示了当前LLM和智能体在超级智能学术研究任务上的能力差距,并凸显了Acadreason的挑战性。

英文摘要:

In recent years, the research focus of large language models (LLMs) and agents has shifted increasingly from demonstrating novel capabilities to complex reasoning and tackling challenging tasks. However, existing evaluations focus mainly on math/code contests or general tasks, while existing multi-domain academic benchmarks lack sufficient reasoning depth, leaving the field without a rigorous benchmark for high-level reasoning. To fill this gap, we introduce the Acadreason benchmark, designed to evaluate the ability of LLMs and agents to acquire and reason over academic knowledge. It consists of 50 expert-annotated academic problems across five high-reasoning domains, including computer science, economics, law, mathematics, and philosophy. All questions are sourced from top-tier publications in recent years and undergo rigorous annotation and quality control to ensure they are both challenging and answerable. We conduct systematic evaluations of over 10 mainstream LLMs and agents. The results show that most LLMs scored below 20 points, with even the cutting-edge GPT-5 achieving only 16 points. While agents achieved higher scores, none exceeded 40 points. This demonstrates the current capability gap between LLMs and agents in super-intelligent academic research tasks and highlights the challenges of Acadreason.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑