arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.15074cs.CLcs.AI
收藏

LLMs已经足够先进了吗?一个面向大型语言模型的具有挑战性的问题求解基准

Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models

Microsoft Research(微软研究院);UC Berkeley(加州大学伯克利分校);IIT Delhi(德里印度理工学院)

Daman Arora, Himanshu Gaurav Singh, Mausam

AI总结:

本文提出JEEBench基准,通过515道IIT JEE-Advanced数学、物理和化学难题评估LLMs的长程推理与问题求解能力,发现即使GPT-4结合多种提示技术性能仍低于40%,并开发事后置信度阈值方法应对倒扣分风险。

摘要:

在过去几年中,大型语言模型(LLMs)在现有推理基准上的性能已显著提升。为此,我们提出了JEEBench,一个用于评估LLMs问题求解能力的更具挑战性的基准数据集。我们从竞争激烈的IIT JEE-Advanced考试中精选了515道具有挑战性的工程类预科数学、物理和化学问题。解决该基准中的问题需要基于深厚领域知识的长程推理。我们对各种开源和专有模型的评估表明,即使使用了自洽性、自我优化和思维链提示等技术,最高性能仍低于40%。最佳模型GPT-4的典型失败模式包括代数运算错误、难以将抽象概念准确转化为数学方程,以及未能检索到相关的领域特定概念。我们还观察到,仅通过提示,GPT-4无法评估错误答案因倒扣分而引入的风险。为此,我们开发了一种基于自洽性的事后置信度阈值方法,以实现有效的响应选择。我们希望这一具有挑战性的基准能指导未来使用LLMs进行问题求解的研究。

英文摘要:

The performance of large language models (LLMs) on existing reasoning benchmarks has significantly improved over the past years. In response, we present JEEBench, a considerably more challenging benchmark dataset for evaluating the problem solving abilities of LLMs. We curate 515 challenging pre-engineering mathematics, physics and chemistry problems from the highly competitive IIT JEE-Advanced exam. Long-horizon reasoning on top of deep in-domain knowledge is essential for solving problems in this benchmark. Our evaluation on various open-source and proprietary models reveals that the highest performance, even after using techniques like self-consistency, self-refinement and chain-of-thought prompting, is less than 40%. The typical failure modes of GPT-4, the best model, are errors in algebraic manipulation, difficulty in grounding abstract concepts into mathematical equations accurately and failure in retrieving relevant domain-specific concepts. We also observe that by mere prompting, GPT-4 is unable to assess risk introduced by negative marking for incorrect answers. For this, we develop a post-hoc confidence-thresholding method over self-consistency, which enables effective response selection. We hope that our challenging benchmark will guide future re-search in problem-solving using LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑