对大规模语言模型(LLMs)进行全面评估的高昂成本需要创建小而有代表性的数据子集(即小型基准),以实现高效的评估同时保留预测保真度。当前的方法基于模型为中心的范式,根据现有模型的集体性能选择基准项目。这些方法受限于前期成本高、无法立即处理新基准(冷启动)以及假设未来模型会共享前代模型的失败模式的脆弱性。在本文中,我们提出了一种新的以项目为中心的基准子集选择方法,认为选择应基于任务项目的内在属性,而不是模型特定的失败模式。我们通过一种新的方法Scales++来实现这种以项目为中心的高效基准方法,其中数据选择基于基准样本的认知需求。实证研究表明,Scales++将前期选择成本降低了超过18倍,同时实现了有竞争力的预测保真度。在Open LLM Leaderboard上,使用仅0.25%的数据子集,我们预测完整基准分数的均方误差为3.2%,在Humanity's Last Exam上,使用2.0%的样本预测完整分数的均方误差为2.9%。我们证明这种以项目为中心的方法可以在不显著降低保真度的情况下更高效地评估模型,同时提供更好的冷启动性能和更可解释的基准测试。
英文摘要:
The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative data subsets (i.e., tiny benchmarks) that enable efficient assessment while retaining predictive fidelity. Current methods for this task operate under a model-centric paradigm, selecting benchmarking items based on the collective performance of existing models. Such approaches are limited by large upfront costs, an inability to immediately handle new benchmarks ("cold-start"), and the fragile assumption that future models will share the failure patterns of their predecessors. In this work, we propose a new item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves, rather than on model-specific failure patterns. We instantiate this item-centric efficient benchmarking approach via a novel method, Scales++, where data selection is based on the cognitive demands of the benchmark samples. Empirically, we show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity. On the Open LLM Leaderboard, using just a 0.25% data subset, we predict full benchmark scores with a 3.2% mean absolute error, and on Humanity's Last Exam we predict full scores with 2.9% mean absolute error using a 2.0% sample. We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation, while also providing better cold-start performance and more interpretable benchmarking.
Beyond ID Embeddings: Process-Grounded Language Modeling for Cognitive Diagnosis
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学)
SuperLocalMemory V3.3: The Living Brain -- Biologically-Inspired Forgetting, Cognitive Quantization, and Multi-Channel Retrieval for Zero-LLM Agent Memory Systems
Do Large Language Models Mirror Cognitive Language Processing?
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);College of Computer and Information Engineering, Tianjin Normal University(天津师范大学计算机与信息工程学院)
Linguistic and Audio Embedding-Based Machine Learning for Alzheimer's Dementia and Mild Cognitive Impairment Detection: Insights from the PROCESS Challenge
Singapore University of Technology and Design(新加坡科技设计大学);Hochschule für Musik Detmold(音乐学院Detmold)
Beyond Discrete Selection: Continuous Embedding Space Optimization for Generative Feature Selection
Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);University of Chinese Academy of Sciences(中国科学院大学);University of Central Florida(中佛罗里达大学);Arizona State University(亚利桑那州立大学)
Frozen Multimodal Embeddings for AI-Assisted Interview Assessment of Personality and Cognitive Ability
Technology Application and Human Resource Development, National Taiwan Normal University(台湾国立台中教育大学技术应用与人力资源发展系);Computer Science and Information Engineering, National Central University(台湾国立中央大学计算机科学与资讯工程系);Institute of Photonic System, National Yang Ming Chiao Tung University(台湾阳明交通大学光电系统研究所)
ActPC-Geom: Towards Scalable Online Neural-Symbolic Learning via Accelerating Active Predictive Coding with Information Geometry & Diverse Cognitive Mechanisms
An Embedding-based Approach to Inconsistency-tolerant Reasoning with Inconsistent Ontologies
School of Computer Science;Engineering, Southeast University Nanjing China;School of Mathematics, Southeast University Nanjing China;School of Modern Posts \& Institute of Modern Posts, Nanjing University of Posts;Engineering, Southeast University;School of Mathematics, Southeast University