Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities. Originally designed to make claims about capabilities (or lack thereof) in fully pretrained models, evaluation benchmarks are now also extensively used to decide between various training choices. Despite this widespread usage, we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful. Here, we define and measure a range of metrics geared towards measuring variance in evaluation benchmarks, including seed variance across initialisations, and monotonicity during training. By studying a large number of models -- both openly available and pretrained from scratch -- we provide empirical estimates for a variety of variance metrics, with considerations and recommendations for practitioners. We also evaluate the utility and tradeoffs of continuous versus discrete performance measures and explore options for better understanding and reducing this variance. We find that simple changes, such as framing choice tasks (like MMLU) as completion tasks, can often reduce variance for smaller scale ($\sim$7B) models, while more involved methods inspired from human testing literature (such as item analysis and item response theory) struggle to meaningfully reduce variance. Overall, our work provides insights into variance in evaluation benchmarks, suggests LM-specific techniques to reduce variance, and more generally encourages practitioners to carefully factor in variance when comparing models.
Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction
Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA(密歇根大学物理系);Michigan Institute for Data and AI in Society, University of Michigan, Ann Arbor, MI 48109, USA(密歇根大学数据与人工智能在社会中的研究所);Independent Researcher, Taiwan(台湾独立研究员)
Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation
Department of Economics & Computer Science(经济与计算机科学系);Institute for Computational and Mathematical Engineering(计算与数学工程研究所);Department of Computer Science(计算机科学系);Department of Aeronautics & Astronautics(航空与航天系)
Accounting for Variance in Machine Learning Benchmarks
Mila(米拉(学习算法研究所));Université de Montréal(蒙特利尔大学);IRIC(免疫和癌症研究所);McGill University(麦吉尔大学);Booz Allen Hamilton(博思艾伦汉密尔顿公司);University of Maryland, Baltimore County(马里兰大学巴尔的摩县分校);École de technologie supérieure(高等技术学院);CIFAR(加拿大高级研究所);Polytechnique Montréal(蒙特利尔理工学院);ElementAI(ElementAI公司);Inria(法国国家信息与自动化研究所)
A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning
Department of Mathematical and Statistical Sciences(数学与统计科学系);Marquette University(马凯特大学);Cornell Ann S. Bowers College of Computing and Information Science(康奈尔大学安·S·博斯计算与信息科学学院);Cornell University(康奈尔大学);Physics Science and Engineering(物理科学与工程);Tongji University(同济大学);School of Computing and Information Systems(计算与信息科学学院);The University of Melbourne(墨尔本大学);Siebel School of Computing and Data Science(希贝尔计算与数据科学学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Divide and Predict: An Architecture for Input Space Partitioning and Enhanced Accuracy
Biocomplexity Institute, University of Virginia(弗吉尼亚大学生物复杂性研究所);Department of Systems and Information Engineering, University of Virginia(弗吉尼亚大学系统与信息工程系);Department of Mathematics, University of Virginia(弗吉尼亚大学数学系)
Quantifying Training Difficulty and Accelerating Convergence in Neural Network-Based PDE Solvers
Hong Kong University of Science and Technology(香港科技大学);Shanghai Jiao Tong University(上海交通大学);The Pennsylvania State University(宾夕法尼亚州立大学);HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute(香港科技大学深港协同创新研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)