arXiv 2503.063782025-03-16cs.AIcs.CLcs.CY

通用量表释放兼具解释力与预测力的 AI 评估

General Scales Unlock AI Evaluation with Explanatory and Predictive Power

Leverhulme Centre for the Future of Intelligence, University of Cambridge（剑桥大学勒沃休姆未来智能中心）；Microsoft Research Asia（微软亚洲研究院）；Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València（瓦伦西亚理工大学瓦伦西亚人工智能研究所）；Department of Engineering, University of Cambridge（剑桥大学工程系）；Department of Psychology, University of Cambridge（剑桥大学心理学系）；The Psychometrics Centre, University of Cambridge（剑桥大学心理测量中心）；Department of Theoretical and Applied Linguistics, University of Cambridge（剑桥大学理论与应用语言学系）；KU Leuven（鲁汶大学）；Educational Testing Service（教育考试服务中心）；Center for Automation and Robotics, Spanish National Research Council（西班牙国家研究委员会自动化与机器人中心）；William & Mary（威廉玛丽学院）；Princeton University（普林斯顿大学）；Carnegie Mellon University（卡内基梅隆大学）

Lexin Zhou, Lorenzo Pacchiardi, Fernando Martínez-Plumed, Katherine M. Collins, Yael Moros-Daval, Seraphina Zhang, Qinlin Zhao, Yitian Huang, Luning Sun, Jonath… 展开作者

Lexin Zhou, Lorenzo Pacchiardi, Fernando Martínez-Plumed, Katherine M. Collins, Yael Moros-Daval, Seraphina Zhang, Qinlin Zhao, Yitian Huang, Luning Sun, Jonathan E. Prunty, Zongqian Li, Pablo Sánchez-García, Kexin Jiang Chen, Pablo A. M. Casares, Jiyun Zu, John Burden, Behzad Mehrbakhsh, David Stillwell, Manuel Cebrian, Jindong Wang, Peter Henderson, Sherry Tongshuang Wu, Patrick C. Kyllonen, Lucy Cheke, Xing Xie, José Hernández-Orallo

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于 18 个评分细则的非饱和通用量表，以解释基准内涵、提取模型能力画像，并在 15 个 LLM、63 项任务上实现优于黑箱方法的实例级分布内外性能预测。

摘要：

确保 AI 的安全有效使用，需要理解并预判其在新颖任务上的表现，这些任务涵盖从前沿科学挑战到转型后的工作场所活动。迄今为止，基准测试一直引导着 AI 的进展，但由于不同任务间的可迁移性较低，它对通用 AI 系统提供的解释力和预测力有限。本文提出用于 AI 评估的通用量表，能够解释常见 AI 基准实际测量的内容，提取 AI 系统的能力画像，并预测其在分布内和分布外的新任务实例上的表现。我们的全自动化方法建立在 18 个新制定的评分细则之上，这些细则将实例需求置于不会饱和的通用量表上。通过对 15 个大型语言模型和 63 项任务的示例说明，考察需求与能力画像释放出强大解释力，揭示了不同基准所表现出的敏感性和特异性，以及模型规模、思维链和蒸馏如何影响知识、元认知与推理。令人惊讶的是，利用这些需求水平可以实现实例级别的高预测力；相较于基于嵌入或微调的黑箱基线预测器，它能给出更优的估计，尤其是在分布外场景（新任务和新基准）中。本文提出的量表、评分细则、测试套件、技术与结果代表了 AI 评估的重要一步，为未来数年可靠部署 AI 提供支撑。（协作平台：https://kinds-of-intelligence-cfi.github.io/ADELE。）

英文摘要：

Ensuring safe and effective use of AI requires understanding and anticipating its performance on novel tasks, from advanced scientific challenges to transformed workplace activities. So far, benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems, given the low transferability across diverse tasks. In this paper, we introduce general scales for AI evaluation that can explain what common AI benchmarks really measure, extract ability profiles of AI systems, and predict their performance for new task instances, in- and out-of-distribution. Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate. Illustrated for 15 large language models and 63 tasks, high explanatory power is unleashed from inspecting the demand and ability profiles, bringing insights on the sensitivity and specificity exhibited by different benchmarks, and how knowledge, metacognition and reasoning are affected by model size, chain-of-thought and distillation. Surprisingly, high predictive power at the instance level becomes possible using these demand levels, providing superior estimates over black-box baseline predictors based on embeddings or finetuning, especially in out-of-distribution settings (new tasks and new benchmarks). The scales, rubrics, battery, techniques and results presented here represent a major step for AI evaluation, underpinning the reliable deployment of AI in the years ahead. (Collaborative platform: https://kinds-of-intelligence-cfi.github.io/ADELE.)

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。