arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11674cs.AI
收藏

基准健康指数:一个系统框架,用于评估大型语言模型的基准测试

Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs

Alibaba Group(阿里巴巴集团);Skylenage

Longyuan Zhu, Hairan Hua, Linlin Miao, Wei Wang, Bing Zhao

AI总结:

本文提出Benchmark Health Index,通过三个维度评估LLM基准测试的健康状况,为评估选择和动态管理提供系统框架。

摘要:

大型语言模型(LLMs)正在迅速发展,但用于衡量这一进步的基准测试却变得越来越不可靠。评分膨胀和选择性报告已削弱了标准基准的权威性,使社区对哪些评估结果仍值得信赖感到困惑。我们介绍了基准健康指数(BHI),这是一个纯粹数据驱动的框架,用于审计评估集,沿三个正交且互补的轴线进行:(1)能力区分,衡量基准如何区分模型性能超出噪声;(2)抗饱和,估计在天花板效应侵蚀分辨率之前剩余的余地;(3)影响,通过采用广度和实践塑造能力量化对学术和工业生态系统的影响。通过提炼2025年91个代表性模型的技术报告中的106个经过验证的基准,我们系统地刻画了评估景观。BHI是第一个量化基准健康状况的宏观框架,为基准选择提供了原则基础,并使下一代评估协议能够实现动态生命周期管理。

英文摘要:

Large Language Models (LLMs) are advancing rapidly, yet the benchmarks used to measure this progress are becoming increasingly unreliable. Score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy. We introduce the Benchmark Health Index (BHI), a pure data-driven framework for auditing evaluation sets along three orthogonal and complementary axes: (1) Capability Discrimination, measuring how sharply a benchmark separates model performance beyond noise; (2) Anti-Saturation, estimating remaining headroom before ceiling effects erode resolution and thus the benchmark's expected longevity; and (3) Impact, quantifying influence across academic and industrial ecosystems via adoption breadth and practice-shaping power. By distilling 106 validated benchmarks from the technical reports of 91 representative models in 2025, we systematically characterize the evaluation landscape. BHI is the first framework to quantify benchmark health at a macro level, providing a principled basis for benchmark selection and enabling dynamic lifecycle management for next-generation evaluation protocols.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑