arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.01719cs.LG
收藏

多任务基准中多样性与稳定性之间的内在权衡

Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarks

Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Tübingen AI Center(图宾根人工智能中心)

Guanhua Zhang, Moritz Hardt

AI总结:

本文借用社会选择理论揭示多任务基准中多样性与稳定性之间的内在权衡,并提出相应度量与近似算法,通过实验证明基准越多样对无关变化越敏感。

摘要:

我们从社会选择理论的视角考察机器学习中的多任务基准。我们将基准与选举系统进行类比,其中模型是候选人,任务是选民。这提示了基数基准系统与序数基准系统之间的区别:前者将数值分数聚合成一个模型排名;后者聚合每个任务的排名。我们将阿罗不可能定理应用于序数基准,以揭示序数系统的固有局限性,特别是它们对纳入无关模型的敏感性。受阿罗定理启发,我们通过实验证明了现有多任务基准中多样性与对无关变化的敏感性之间存在强烈的权衡。我们的结果基于我们引入的新的多样性和敏感性定量度量。敏感性量化了任务的无关变化对基准的影响。多样性刻画了模型排名在不同任务之间的分歧程度。我们为这两个度量开发了高效的近似算法,因为精确计算在计算上具有挑战性。通过在七个基数基准和十一个序数基准上的大量实验,我们证明了多样性与稳定性之间存在明显的权衡:多任务基准越多样,它对琐碎变化就越敏感。此外,我们表明现有基准的聚合排名在无关变化下高度不稳定。代码和数据可在 https://socialfoundations.github.io/benchbench/ 获取。

英文摘要:

We examine multi-task benchmarks in machine learning through the lens of social choice theory. We draw an analogy between benchmarks and electoral systems, where models are candidates and tasks are voters. This suggests a distinction between cardinal and ordinal benchmark systems. The former aggregate numerical scores into one model ranking; the latter aggregate rankings for each task. We apply Arrow's impossibility theorem to ordinal benchmarks to highlight the inherent limitations of ordinal systems, particularly their sensitivity to the inclusion of irrelevant models. Inspired by Arrow's theorem, we empirically demonstrate a strong trade-off between diversity and sensitivity to irrelevant changes in existing multi-task benchmarks. Our result is based on new quantitative measures of diversity and sensitivity that we introduce. Sensitivity quantifies the impact that irrelevant changes to tasks have on a benchmark. Diversity captures the degree of disagreement in model rankings across tasks. We develop efficient approximation algorithms for both measures, as exact computation is computationally challenging. Through extensive experiments on seven cardinal benchmarks and eleven ordinal benchmarks, we demonstrate a clear trade-off between diversity and stability: The more diverse a multi-task benchmark, the more sensitive to trivial changes it is. Additionally, we show that the aggregated rankings of existing benchmarks are highly unstable under irrelevant changes. The codes and data are available at https://socialfoundations.github.io/benchbench/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑