arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.01869cs.CLcs.AI
收藏

超越准确率:评估大语言模型推理行为的综述

Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey

LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心)

Philipp Mondorf, Barbara Plank

AI总结:

本文综述了超越准确率指标的大语言模型推理行为评估研究,指出LLMs主要依赖表层模式而非深层推理,并呼吁进一步区分人类与模型推理差异。

摘要:

大语言模型(LLMs)最近在涉及推理的任务上表现出令人瞩目的性能,引发了关于这些模型是否具备与人类相似推理能力的激烈讨论。然而,尽管取得了这些成功,LLMs推理能力的深度仍不确定。这种不确定性部分源于当前研究主要关注任务性能,并通过浅层的准确率指标来衡量,而缺乏对模型推理行为的深入考察。本文旨在弥补这一不足,系统回顾了超越任务准确率的研究,提供对模型推理过程更深入的见解。此外,我们综述了评估LLMs推理行为的常用方法,强调当前趋势以及朝向更细致推理分析的努力。我们的综述表明,LLMs倾向于依赖训练数据中的表层模式和相关性,而非复杂的推理能力。此外,我们指出需要进一步研究以厘清人类推理与基于LLM的推理之间的关键差异。通过本综述,我们希望揭示LLMs内部复杂的推理过程。

英文摘要:

Large language models (LLMs) have recently shown impressive performance on tasks involving reasoning, leading to a lively debate on whether these models possess reasoning capabilities similar to humans. However, despite these successes, the depth of LLMs' reasoning abilities remains uncertain. This uncertainty partly stems from the predominant focus on task performance, measured through shallow accuracy metrics, rather than a thorough investigation of the models' reasoning behavior. This paper seeks to address this gap by providing a comprehensive review of studies that go beyond task accuracy, offering deeper insights into the models' reasoning processes. Furthermore, we survey prevalent methodologies to evaluate the reasoning behavior of LLMs, emphasizing current trends and efforts towards more nuanced reasoning analyses. Our review suggests that LLMs tend to rely on surface-level patterns and correlations in their training data, rather than on sophisticated reasoning abilities. Additionally, we identify the need for further research that delineates the key differences between human and LLM-based reasoning. Through this survey, we aim to shed light on the complex reasoning processes within LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑