arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.03597cs.CLcs.LGstat.ML
收藏

语言模型基准的脆弱性:它们是否准确反映真实的 LLM 性能?

The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?

DataLabs, United We Care(DataLabs,United We Care)

Sourav Banerjee, Ayushi Agarwal, Eishkaran Singh

AI总结:

本文系统剖析 NLP 评估框架在基准钻营、数据污染和偏差方面的脆弱性,并提出构建抗操纵、低污染、面向领域任务的动态评估框架。

摘要:

对大型语言模型(LLM)排行榜排名的追逐造成了一个根本性悖论:模型擅长标准化测试,却未能展现真正的语言理解与适应能力。我们对 NLP 评估框架的系统分析揭示,从基础指标到 GLUE、MMLU 等复杂基准,整个评估链条中普遍存在脆弱性。这些脆弱性表现为基准钻营、数据集污染和评估偏差,造成语言理解能力取得进展的虚假印象。通过广泛综述当代评估方法,我们发现静态基准设计、人工评估协议以及 LLM-as-judge 框架均存在显著局限,这些问题共同削弱了当前性能评估的可靠性。随着 LLM 能力演进、既有基准趋于冗余,我们为新型评估方法奠定基础:这些方法应能抵抗操纵、最大限度减少数据污染并评估领域特定任务。这要求框架能够动态调整,以解决当前局限,并更准确地反映 LLM 的真实性能。

英文摘要:

The pursuit of leaderboard rankings in Large Language Models (LLMs) has created a fundamental paradox: models excel at standardized tests while failing to demonstrate genuine language understanding and adaptability. Our systematic analysis of NLP evaluation frameworks reveals pervasive vulnerabilities across the evaluation spectrum, from basic metrics to complex benchmarks like GLUE and MMLU. These vulnerabilities manifest through benchmark exploitation, dataset contamination, and evaluation bias, creating a false perception of progress in language understanding capabilities. Through extensive review of contemporary evaluation approaches, we identify significant limitations in static benchmark designs, human evaluation protocols, and LLM-as-judge frameworks, all of which compromise the reliability of current performance assessments. As LLM capabilities evolve and existing benchmarks become redundant, we lay the groundwork for new evaluation methods that resist manipulation, minimize data contamination, and assess domain-specific tasks. This requires frameworks that are adapted dynamically, addressing current limitations and providing a more accurate reflection of LLM performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑