这些大语言模型基准测试是否一致?用BenchBench修正基准评估
Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench
AI总结:
针对大语言模型基准一致性测试(BAT)缺乏标准化流程的问题,研究分析40余个主流基准,提出BAT最佳实践,并推出Python包BenchBench及元基准排行榜,提升基准评估的鲁棒性与有效性。
摘要:
语言模型(LM)的最新进展催生了多个旨在评估模型通用能力的基准测试。然而,一项关键任务是评估这些基准本身的有效性。这通常通过基准一致性测试(BAT)完成,即使用某种一致性指标(如秩相关),将新基准与已确立的基准进行对比验证。尽管BAT对基准构建者和使用者至关重要,但目前尚无此类一致性测试的标准化流程。这一缺陷可能导致无效结论,加剧对基准的不信任,还会破坏人们合理选择适用基准的能力。通过分析40多个主流基准,我们证明了一些被忽视的方法学选择会显著影响BAT结果,可能削弱结论的有效性。为解决这些不一致问题,我们提出了一套BAT最佳实践,并证明采用这些方法可大幅提升BAT的鲁棒性和有效性。为推动应用并促进未来研究,我们推出了用于BAT的Python包BenchBench,还发布了BenchBench排行榜——一个旨在通过同类基准对比来评估各基准的元基准。我们的研究结果凸显了标准化BAT的必要性,以确保在不断发展的语言模型研究领域中,基准评估的鲁棒性和有效性。BenchBench工具包:github.com/IBM/BenchBench;排行榜:hf.co/spaces/IBM/BenchBench
英文摘要:
Recent advancements in Language Models (LMs) have catalyzed the creation of multiple benchmarks, designed to assess these models' general capabilities. A crucial task, however, is assessing the validity of the benchmarks themselves. This is most commonly done via Benchmark Agreement Testing (BAT), where new benchmarks are validated against established ones using some agreement metric (e.g., rank correlation). Despite the crucial role of BAT for benchmark builders and consumers, there are no standardized procedures for such agreement testing. This deficiency can lead to invalid conclusions, fostering mistrust in benchmarks and upending the ability to properly choose the appropriate benchmark to use. By analyzing over 40 prominent benchmarks, we demonstrate how some overlooked methodological choices can significantly influence BAT results, potentially undermining the validity of conclusions. To address these inconsistencies, we propose a set of best practices for BAT and demonstrate how utilizing these methodologies greatly improves BAT robustness and validity. To foster adoption and facilitate future research,, we introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers. Our findings underscore the necessity for standardized BAT, ensuring the robustness and validity of benchmark evaluations in the evolving landscape of language model research. BenchBench Package: github.com/IBM/BenchBench Leaderboard: hf.co/spaces/IBM/BenchBench
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。