arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.16711cs.SEcs.CL
收藏

基准测试作为显微镜:对模型度量学的呼吁

Benchmarks as Microscopes: A Call for Model Metrology

University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);University of Chicago(芝加哥大学);University of Washington(华盛顿大学);Kempner Institute at Harvard University(哈佛大学坎普纳研究所)

Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, Naomi Saphra

AI总结:

针对静态基准饱和导致语言模型能力评估不可靠的问题,提出模型度量学这一新学科,通过动态评估和建立从业者社区来预测部署性能,为AI发展提供可靠指标。

摘要:

现代语言模型(LMs)在能力评估方面提出了新的挑战。静态基准不可避免地会饱和,无法为基于LM的系统提供部署容差方面的置信度,但开发者仍然声称他们的模型基于这些有缺陷的指标具备了推理或开放域语言理解等泛化特征。LMs的科学和实践需要一种新的基准测试方法,通过动态评估来衡量特定能力。为了对我们的指标有信心,我们需要一门新的模型度量学学科——专注于如何生成预测部署下性能的基准测试。基于我们的评估标准,我们概述了如何建立一个由模型度量学从业者组成的社区——专注于构建工具和研究如何衡量系统能力——这是满足这些需求并为AI讨论增加清晰度的最佳方式。

英文摘要:

Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-based systems, but developers nonetheless claim that their models have generalized traits such as reasoning or open-domain language understanding based on these flawed metrics. The science and practice of LMs requires a new approach to benchmarking which measures specific capabilities with dynamic assessments. To be confident in our metrics, we need a new discipline of model metrology -- one which focuses on how to generate benchmarks that predict performance under deployment. Motivated by our evaluation criteria, we outline how building a community of model metrology practitioners -- one focused on building tools and studying how to measure system capabilities -- is the best way to meet these needs to and add clarity to the AI discussion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑