arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11898cs.CL
收藏

基准幻觉:大语言模型之间的分歧及其科学后果

Benchmark Illusion: Disagreement among LLMs and Its Scientific Consequences

Department of Political Science, Purdue University(政治学系,普渡大学);Kellogg School of Management, Center for Science of Science and Innovation, Northwestern Innovation Institute, Ryan Institute on Complexity, and McCormick School of Engineering(凯洛格管理学院、科学与创新中心、西北创新研究所、复杂性研究院以及麦科马克工程学院)

Eddie Yang, Dashun Wang

AI总结:

研究揭示大语言模型在基准测试中存在显著分歧,影响科学可重复性,指出模型选择对研究结果有重要影响。

摘要:

基准测试是衡量和信任大语言模型(LLM)进展的基础。然而,我们的分析表明,看似在基准准确率上的收敛可能掩盖了深层次的认知分歧。使用两个主要的推理基准测试——MMLU-Pro和GPQA——我们发现,即使LLM在准确率上相当,它们在16-66%的项目上仍然存在分歧,而在顶尖前沿模型之间,这种分歧达到16-38%。这些差异表明不同LLM有不同的错误特征。当这些模型用于科学数据标注和推理时,其隐藏的分歧会传播到研究结果中:在重新分析教育和政治科学已发表的研究时,切换标注模型可以将估计的治疗效应改变超过80%,在某些情况下甚至会改变其符号。这些发现共同展示了基准幻觉,即相等的准确率可能掩盖分歧,模型选择成为影响科学可重复性的一个隐含但重要的变量。

英文摘要:

Benchmarks underpin how progress in large language models (LLMs) is measured and trusted. Yet our analyses reveal that apparent convergence in benchmark accuracy can conceal deep epistemic divergence. Using two major reasoning benchmarks - MMLU-Pro and GPQA - we show that LLMs achieving comparable accuracy still disagree on 16-66% of items, and 16-38% among top-performing frontier models. These discrepancies suggest distinct error profiles for different LLMs. When such models are used for scientific data annotation and inference, their hidden disagreements propagate into research results: in re-analyses of published studies in education and political science, switching the annotation model can change estimated treatment effects by more than 80%, and in some cases reverses their sign. Together, these findings illustrate a benchmark illusion, where equal accuracy may conceal disagreement, with model choice becoming a hidden yet consequential variable for scientific reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑