We examine multi-task benchmarks in machine learning through the lens of social choice theory. We draw an analogy between benchmarks and electoral systems, where models are candidates and tasks are voters. This suggests a distinction between cardinal and ordinal benchmark systems. The former aggregate numerical scores into one model ranking; the latter aggregate rankings for each task. We apply Arrow's impossibility theorem to ordinal benchmarks to highlight the inherent limitations of ordinal systems, particularly their sensitivity to the inclusion of irrelevant models. Inspired by Arrow's theorem, we empirically demonstrate a strong trade-off between diversity and sensitivity to irrelevant changes in existing multi-task benchmarks. Our result is based on new quantitative measures of diversity and sensitivity that we introduce. Sensitivity quantifies the impact that irrelevant changes to tasks have on a benchmark. Diversity captures the degree of disagreement in model rankings across tasks. We develop efficient approximation algorithms for both measures, as exact computation is computationally challenging. Through extensive experiments on seven cardinal benchmarks and eleven ordinal benchmarks, we demonstrate a clear trade-off between diversity and stability: The more diverse a multi-task benchmark, the more sensitive to trivial changes it is. Additionally, we show that the aggregated rankings of existing benchmarks are highly unstable under irrelevant changes. The codes and data are available at https://socialfoundations.github.io/benchbench/.
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
Georgia Tech(佐治亚理工学院);Cornell University(康奈尔大学);University of Limerick(利默里克大学);Sakana AI;SynthLabs;Reka AI;Stanford University(斯坦福大学);Eleuther AI;IBM;Cohere for AI Community;University of Bologna(博洛尼亚大学);Cognizant AI Labs
A Trajectory-Based Bayesian Approach to Multi-Objective Hyperparameter Optimization with Epoch-Aware Trade-Offs
Department of Industrial Systems Engineering and Management, National University of Singapore(新加坡国立大学工业系统工程与管理系);Academy of Mathematics and System Sciences, Chinese Academy of Sciences(中国科学院数学系统科学 academy);School of Mathematical Sciences, University of Chinese Academy of Sciences(中国科学院大学数学科学学院)