arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.10229cs.LGcs.AI
收藏

量化评估基准中的方差

Quantifying Variance in Evaluation Benchmarks

Meta;UCL(伦敦大学学院);Stanford University(斯坦福大学);Cohere(科here公司)

Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, Dieuwke Hupkes

AI总结:

本文系统定义并测量 LLM 评估基准中的初始化种子方差与训练单调性等指标,发现将 MMLU 等选择题改为补全可降低约 7B 模型方差,而项目反应理论等复杂方法效果有限。

摘要:

评估基准是衡量大型语言模型(LLM)能力并推动这些能力进步的基石。评估基准最初旨在对完全预训练模型所具备(或不具备)的能力作出论断,如今也被广泛用于在各种训练选择之间作出决策。尽管应用如此广泛,我们却很少量化评估基准中的方差,而这种方差决定了性能差异是否具有意义。在本文中,我们定义并测量了一系列旨在衡量评估基准方差的指标,包括不同初始化之间的随机种子方差,以及训练过程中的单调性。通过研究大量模型——既有公开可用的模型,也有从头开始预训练的模型——我们为多种方差指标提供了经验估计,并为从业者给出了考量因素和建议。我们还评估了连续与离散性能度量的效用和权衡,并探索了更好地理解和降低这种方差的方案。我们发现,简单的改变,例如将选择题任务(如 MMLU)构造为补全任务,通常可以降低较小规模(约 7B)模型的方差;而受人类测试文献启发的更复杂方法(如项目分析和项目反应理论)则难以有意义地降低方差。总体而言,我们的工作提供了关于评估基准中方差的洞见,提出了降低方差的 LM 专用技术,并更广泛地鼓励从业者在比较模型时谨慎考虑方差因素。

英文摘要:

Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities. Originally designed to make claims about capabilities (or lack thereof) in fully pretrained models, evaluation benchmarks are now also extensively used to decide between various training choices. Despite this widespread usage, we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful. Here, we define and measure a range of metrics geared towards measuring variance in evaluation benchmarks, including seed variance across initialisations, and monotonicity during training. By studying a large number of models -- both openly available and pretrained from scratch -- we provide empirical estimates for a variety of variance metrics, with considerations and recommendations for practitioners. We also evaluate the utility and tradeoffs of continuous versus discrete performance measures and explore options for better understanding and reducing this variance. We find that simple changes, such as framing choice tasks (like MMLU) as completion tasks, can often reduce variance for smaller scale ($\sim$7B) models, while more involved methods inspired from human testing literature (such as item analysis and item response theory) struggle to meaningfully reduce variance. Overall, our work provides insights into variance in evaluation benchmarks, suggests LM-specific techniques to reduce variance, and more generally encourages practitioners to carefully factor in variance when comparing models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑