arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.05692cs.CL
收藏

超越准确率的数学推理能力评估

Evaluating Mathematical Reasoning Beyond Accuracy

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);Generative AI Research Lab (GAIR)(生成式人工智能研究实验室(GAIR))

Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, Pengfei Liu

AI总结:

针对现有大语言模型数学评估仅关注最终答案、忽略推理步骤质量的问题,提出ReasonEval评估方法,通过有效性和冗余性衡量推理质量,实验证明其性能优于基线,且可用于数据筛选,相关资源已开源。

摘要:

大语言模型(LLMs)在数学任务中的排行榜持续更新,但大多数评估仅关注最终结果,忽略了中间步骤的质量。这种疏忽可能掩盖潜在问题,例如推理过程中的逻辑错误或不必要步骤。为了衡量最终答案准确率之外的推理能力,我们提出了ReasonEval(推理评估),一种用于评估推理步骤质量的新方法。ReasonEval采用有效性和冗余性来表征推理质量,并配套使用大语言模型自动评估这两项指标。我们探索了基于大语言模型的评估器的不同设计方案,并通过实验证明,当ReasonEval由具备强数学知识的基础模型实例化、并使用高质量标注数据训练时,在元评估数据集上的表现始终优于基线方法。我们还强调了ReasonEval强大的泛化能力。通过使用ReasonEval评估专门用于数学领域的大语言模型,我们发现,对于有挑战性的数学问题,最终答案准确率的提升并不一定能保证推理步骤整体质量的提升。此外,我们观察到ReasonEval可以在数据筛选中发挥重要作用。我们开源了性能最优的模型、元评估脚本和所有评估结果,以推动未来的研究。

英文摘要:

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs validity and redundancy to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. We explore different design options for the LLM-based evaluators and empirically demonstrate that ReasonEval, when instantiated with base models possessing strong mathematical knowledge and trained with high-quality labeled data, consistently outperforms baseline methods in the meta-evaluation datasets. We also highlight the strong generalization capabilities of ReasonEval. By utilizing ReasonEval to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We open-source the best-performing model, meta-evaluation script, and all evaluation results to facilitate future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑