arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2307.13692cs.CLcs.LG
收藏

ARB:大型语言模型的高级推理基准

ARB: Advanced Reasoning Benchmark for Large Language Models

DuckAI;Georgia Tech(佐治亚理工学院);ETH Zürich(苏黎世联邦理工学院);Nomos AI;Stanford University Center for Legal Informatics(斯坦福大学法律信息学中心);MILA(MILA(蒙特利尔学习算法研究所))

Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, Aran Komatsuzaki

AI总结:

针对现有基准因LLM高分而失效的问题,提出涵盖多领域的高级推理基准ARB,评估显示当前模型在困难任务上得分低于50%,并引入基于评分标准的自评方法及人工验证。

摘要:

大型语言模型(LLMs)在各种定量推理和知识基准上表现出了卓越的性能。然而,随着 LLMs 的得分越来越高,许多此类基准正在失去效用,尽管它们尚未在这些领域达到专家水平。我们引入了 ARB,这是一个由多领域高级推理问题组成的新颖基准。ARB 提供了比以往基准更具挑战性的测试,涵盖数学、物理、生物、化学和法律领域的问题。作为 ARB 的一个子集,我们引入了一组具有挑战性的数学和物理问题,需要高级符号推理和领域知识。我们在 ARB 上评估了 GPT-4 和 Claude 等近期模型,并证明当前模型在要求更高的任务上得分远低于 50%。为了提升自动和辅助评估能力,我们引入了一种基于评分标准的评估方法,允许 GPT-4 对其自身的中间推理步骤进行评分。此外,我们对 ARB 的符号子集进行了人工评估,发现标注者与 GPT-4 评分标准评估得分之间具有很高的一致性。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable performance on various quantitative reasoning and knowledge benchmarks. However, many of these benchmarks are losing utility as LLMs get increasingly high scores, despite not yet reaching expert performance in these domains. We introduce ARB, a novel benchmark composed of advanced reasoning problems in multiple fields. ARB presents a more challenging test than prior benchmarks, featuring problems in mathematics, physics, biology, chemistry, and law. As a subset of ARB, we introduce a challenging set of math and physics problems which require advanced symbolic reasoning and domain knowledge. We evaluate recent models such as GPT-4 and Claude on ARB and demonstrate that current models score well below 50% on more demanding tasks. In order to improve both automatic and assisted evaluation capabilities, we introduce a rubric-based evaluation approach, allowing GPT-4 to score its own intermediate reasoning steps. Further, we conduct a human evaluation of the symbolic subset of ARB, finding promising agreement between annotators and GPT-4 rubric evaluation scores.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑