arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.07575cs.AIcs.LG
收藏

基准测试已失效——勿让AI自任裁判

Benchmarking is Broken -- Don't Let AI be its Own Judge

Princeton University(普林斯顿大学);CISPA Helmholtz Center for Information Security(CISPA海德堡信息安全中心);Michigan State University(密歇根州立大学);Ohio State University(俄亥俄州立大学);Brown University(布朗大学);Massachusetts Institute of Technology(麻省理工学院);University of California, Los Angeles(加州大学洛杉矶分校);University of Tübingen(图宾根大学);Old Dominion University(旧 Dominion 大学);Technical University of Munich(慕尼黑技术大学);Cornell University(康奈尔大学);Forest AI(森林AI)
Zerui Cheng, Stella Wohnig, Ruchika Gupta, Samiul Alam, Tassallah Abdullahi, João Alves Ribeiro, Christian Nielsen-Garcia, Saif Mir, Siran Li, Jason Orender, Se… 展开作者

Zerui Cheng, Stella Wohnig, Ruchika Gupta, Samiul Alam, Tassallah Abdullahi, João Alves Ribeiro, Christian Nielsen-Garcia, Saif Mir, Siran Li, Jason Orender, Seyed Ali Bahrainian, Daniel Kirste, Aaron Gokaslan, Mikołaj Glinka, Carsten Eickhoff, Ruben Wolff

AI总结:

针对当前AI基准测试因数据污染和选择性报告而失效的问题,本文提出建立统一、实时、质量受控的评估框架,并介绍社区治理的PeerBench蓝图,以恢复评估诚信。

摘要:

AI的迅猛崛起及其快速扩张的市值,既带来了变革性的机遇,也带来了严峻的挑战。其中首要的是迫切需要一种新的、统一的、可信评估范式,因为当前的基准测试日益暴露出关键弱点。数据污染和模型开发者选择性报告等问题助长了炒作,而数据质量控制不足可能导致有偏见的评估,即使是无意的,也可能偏向特定方法。随着大量参与者涌入AI领域,这种评估的“狂野西部”使得区分真正进展与夸大其词变得异常困难。这种模糊性模糊了科学信号,侵蚀了公众信心,正如不受约束的声明会破坏依赖穆迪等机构可信监督的金融市场稳定一样。在高风险的人类考试(如SAT、GRE)中,大量努力被用于确保公平性和可信度;为何在评估AI时,尤其是考虑到其深远的社会影响,却要降低标准?这篇立场论文认为,当前自由放任的方法是不可持续的。我们主张,真正可持续的AI进步需要范式转变:一个统一的、实时的、质量受控的基准测试框架,其稳健性源于构建本身,而非仅仅依靠礼貌和善意。为此,我们剖析了破坏当今AI评估的系统性缺陷,提炼了新一代评估的基本要求,并介绍了PeerBench(其原型实现见https://www.peerbench.ai/),这是一个社区治理、有监考的评估蓝图,通过密封执行、滚动更新的题库和延迟透明度来体现这一范式。我们的目标是为恢复诚信、提供真正可信的AI进展衡量标准的评估铺平道路。

英文摘要:

The meteoric rise of AI, with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need for a new, unified paradigm for trustworthy evaluation, as current benchmarks increasingly reveal critical vulnerabilities. Issues like data contamination and selective reporting by model developers fuel hype, while inadequate data quality control can lead to biased evaluations that, even if unintentionally, may favor specific approaches. As a flood of participants enters the AI space, this "Wild West" of assessment makes distinguishing genuine progress from exaggerated claims exceptionally difficult. Such ambiguity blurs scientific signals and erodes public confidence, much as unchecked claims would destabilize financial markets reliant on credible oversight from agencies like Moody's. In high-stakes human examinations (e.g., SAT, GRE), substantial effort is devoted to ensuring fairness and credibility; why settle for less in evaluating AI, especially given its profound societal impact? This position paper argues that the current laissez-faire approach is unsustainable. We contend that true, sustainable AI advancement demands a paradigm shift: a unified, live, and quality-controlled benchmarking framework robust by construction, not by mere courtesy and goodwill. To this end, we dissect the systemic flaws undermining today's AI evaluation, distill the essential requirements for a new generation of assessments, and introduce PeerBench (with its prototype implementation at https://www.peerbench.ai/), a community-governed, proctored evaluation blueprint that embodies this paradigm through sealed execution, item banking with rolling renewal, and delayed transparency. Our goal is to pave the way for evaluations that can restore integrity and deliver genuinely trustworthy measures of AI progress.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑