arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26384cs.AIcs.LG
收藏

Scales++: 一种计算高效的评估子集选择方法,基于认知尺度嵌入

Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings

Thomson Reuters Foundational Research(汤姆森路透基础研究);University of Oxford(牛津大学);Imperial College London(帝国理工学院伦敦分校)

Andrew M. Bean, Nabeel Seedat, Shengzhuang Chen, Jonathan Richard Schwarz

AI总结:

本文提出了一种基于任务项目内在属性的评估子集选择方法Scales++,通过减少预选成本并保持预测保真度,提高了大规模语言模型的评估效率,同时提升了冷启动性能和可解释性。

摘要:

对大规模语言模型(LLMs)进行全面评估的高昂成本需要创建小而有代表性的数据子集(即小型基准),以实现高效的评估同时保留预测保真度。当前的方法基于模型为中心的范式,根据现有模型的集体性能选择基准项目。这些方法受限于前期成本高、无法立即处理新基准(冷启动)以及假设未来模型会共享前代模型的失败模式的脆弱性。在本文中,我们提出了一种新的以项目为中心的基准子集选择方法,认为选择应基于任务项目的内在属性,而不是模型特定的失败模式。我们通过一种新的方法Scales++来实现这种以项目为中心的高效基准方法,其中数据选择基于基准样本的认知需求。实证研究表明,Scales++将前期选择成本降低了超过18倍,同时实现了有竞争力的预测保真度。在Open LLM Leaderboard上,使用仅0.25%的数据子集,我们预测完整基准分数的均方误差为3.2%,在Humanity's Last Exam上,使用2.0%的样本预测完整分数的均方误差为2.9%。我们证明这种以项目为中心的方法可以在不显著降低保真度的情况下更高效地评估模型,同时提供更好的冷启动性能和更可解释的基准测试。

英文摘要:

The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative data subsets (i.e., tiny benchmarks) that enable efficient assessment while retaining predictive fidelity. Current methods for this task operate under a model-centric paradigm, selecting benchmarking items based on the collective performance of existing models. Such approaches are limited by large upfront costs, an inability to immediately handle new benchmarks ("cold-start"), and the fragile assumption that future models will share the failure patterns of their predecessors. In this work, we propose a new item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves, rather than on model-specific failure patterns. We instantiate this item-centric efficient benchmarking approach via a novel method, Scales++, where data selection is based on the cognitive demands of the benchmark samples. Empirically, we show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity. On the Open LLM Leaderboard, using just a 0.25% data subset, we predict full benchmark scores with a 3.2% mean absolute error, and on Humanity's Last Exam we predict full scores with 2.9% mean absolute error using a 2.0% sample. We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation, while also providing better cold-start performance and more interpretable benchmarking.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑