arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00710cs.AI
收藏

从更少中学习更多:解锁内部表示以实现基准压缩

Learning More from Less: Unlocking Internal Representations for Benchmark Compression

School of Computer Science, Beijing Institute of Technology(北京理工大学计算机科学学院)

Yueqi Zhang, Jin Hu, Shaoxiong Feng, Peiwen Yuan, Xinglin Wang, Yiwei Li, Jiayi Shi, Chuyi Tan, Ji Zhang, Boyuan Pan, Yao Hu, Kan Li

AI总结:

提出RepCore方法,通过对齐异构隐藏状态构建代表性核心集,仅需少量源模型即可精确估计大语言模型在完整基准上的性能,显著优于基于输出的基线方法。

摘要:

评估大语言模型(LLM)的高昂成本促使需要全规模基准测试的高效替代方案。主流方法通过识别少量核心项来近似完整基准性能。然而,现有方法必须从多个源模型的响应模式中估计可靠的项轮廓,当源池较小时,这在统计上变得不稳定。这种依赖性对于历史评估数据极少的新发布基准尤为受限。我们认为离散的正确性标签是模型决策过程的有损视图,未能捕捉隐藏状态中编码的信息。为解决此问题,我们引入RepCore,它将异构隐藏状态对齐到统一潜在空间以构建代表性核心集。使用这些子集进行性能外推,RepCore在仅使用十个源模型时即可实现精确的估计精度。在五个基准和超过200个模型上的实验表明,在排序相关性和估计精度方面,RepCore始终优于基于输出的基线。频谱分析进一步表明,对齐的表示包含反映广泛响应倾向和任务特定推理模式的可分离成分。

英文摘要:

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model's decision process and fail to capture information encoded in hidden states. To address this, we introduce RepCore, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, RepCore achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑