我们的测试是否过多?面向高效评估大语言模型的基准压缩研究综述
Are We Testing Too Much? A Survey of Benchmark Compression for Efficient Evaluation of Large Language Models
AI总结:
该综述回顾119篇论文,构建大语言模型基准压缩分类体系,对比各类方法的特性,为研究人员提供高效可靠评估LLM的方法选择指南。
摘要:
基准是评估和改进大语言模型(LLMs)的关键,但许多广泛使用的基准存在冗余,导致评估成本过高。基准压缩通过部分评估来估算完整基准的分数和排名,从而降低成本。自2024年以来,该领域研究发展迅速,但据我们所知,尚无综述系统整理其方法和发现。我们回顾了119篇论文,解释了方法如何选择评估内容、恢复完整基准结果并验证其可靠性。基于此分析,我们构建了涵盖单个基准内压缩、跨基准套件压缩、自适应评估期间压缩,以及冗余与可靠性研究的分类体系,同时对比了各类方法的信息需求、总成本、可靠性检查及局限性。这些方法通过选择代表性子集、使用更廉价或可重复利用的信息、进行自适应测试来降低评估成本,但存在覆盖不足、估算不准确、序列延迟等风险。该分类体系和实用指南可帮助研究人员选择、验证和部署相关方法,以实现更高效、可靠的LLM评估。
英文摘要:
Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。