arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.03461cs.LGcs.CL
收藏

大型语言模型基准测试是否检验了可靠性?

Do Large Language Model Benchmarks Test Reliability?

MIT(麻省理工学院)

Joshua Vendrow, Edward Vendrow, Sara Beery, Aleksander Madry

AI总结:

本文揭示现有LLM基准测试因标签错误而难以衡量可靠性,提出铂金基准测试概念并修订15个流行基准,发现前沿模型在简单任务上仍存在系统性失败。

摘要:

在部署大型语言模型(LLMs)时,重要的是确保这些模型不仅具备能力,而且可靠。目前已创建了许多基准测试来跟踪LLMs不断增长的能力,然而却缺乏对衡量其可靠性的类似关注。为了理解这一差距的潜在后果,我们研究了当前基准测试在多大程度上能够量化模型可靠性。我们发现,普遍存在的标签错误可能损害这些评估,掩盖了残留的模型失败并隐藏了不可靠行为。受可靠性评估中这一差距的启发,我们随后提出了所谓的铂金基准测试(platinum benchmarks)概念,即经过精心整理以最小化标签错误和歧义的基准测试。作为构建此类基准测试的首次尝试,我们修订了来自十五个现有流行基准测试的示例。我们在这些铂金基准测试上评估了广泛的模型,并发现前沿LLMs确实在诸如小学水平数学文字题之类的简单任务上仍然表现出失败。进一步分析这些失败揭示了此前未被识别的、前沿模型持续面临困难的问题模式。我们在https://github.com/MadryLab/platinum-benchmarks提供代码。

英文摘要:

When deploying large language models (LLMs), it is important to ensure that these models are not only capable, but also reliable. Many benchmarks have been created to track LLMs' growing capabilities, however there has been no similar focus on measuring their reliability. To understand the potential ramifications of this gap, we investigate how well current benchmarks quantify model reliability. We find that pervasive label errors can compromise these evaluations, obscuring lingering model failures and hiding unreliable behavior. Motivated by this gap in the evaluation of reliability, we then propose the concept of so-called platinum benchmarks, i.e., benchmarks carefully curated to minimize label errors and ambiguity. As a first attempt at constructing such benchmarks, we revise examples from fifteen existing popular benchmarks. We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks such as elementary-level math word problems. Analyzing these failures further reveals previously unidentified patterns of problems on which frontier models consistently struggle. We provide code at https://github.com/MadryLab/platinum-benchmarks

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑