arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.01867cs.LG
收藏

数据集难度与归纳偏置的作用

Dataset Difficulty and the Role of Inductive Bias

McGill University(麦吉尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Amazon, Alexa AI(亚马逊Alexa AI);MosaicML;Google DeepMind(谷歌DeepMind)

Devin Kwok, Nikhil Anand, Jonathan Frankle, Gintare Karolina Dziugaite, David Rolnick

AI总结:

本研究系统比较了不同评分方法在多运行和模型架构间的一致性,发现评分存在噪声且与单一难度概念强相关,并基于统计遗传学开发了一种模型架构指纹识别方法,为数据集评分实践提供了指导与基线。

摘要:

受数据集剪枝和缺陷识别目标的推动,越来越多的方法被开发出来用于对数据集中的单个样本进行评分。这些方法我们称之为“样本难度评分”,通常用于对样本进行排名或分类,但不同训练运行、评分方法和模型架构之间排名的一致性通常是未知的。为了确定样本排名如何因这些随机和受控效应而变化,我们在一系列运行和模型架构上系统地比较了不同的评分公式。我们发现这些评分在很大程度上具有以下特征:它们在模型的单次运行中存在噪声,与单一难度概念强相关,并揭示了从对某些模型架构的归纳偏置高度敏感到不敏感的样本。借鉴统计遗传学,我们开发了一种使用少数敏感样本对模型架构进行指纹识别的简单方法。这些发现指导从业者最大化其评分的一致性(例如通过选择适当的评分方法、运行次数和样本子集),并为未来评估评分建立了全面的基线。

英文摘要:

Motivated by the goals of dataset pruning and defect identification, a growing body of methods have been developed to score individual examples within a dataset. These methods, which we call "example difficulty scores", are typically used to rank or categorize examples, but the consistency of rankings between different training runs, scoring methods, and model architectures is generally unknown. To determine how example rankings vary due to these random and controlled effects, we systematically compare different formulations of scores over a range of runs and model architectures. We find that scores largely share the following traits: they are noisy over individual runs of a model, strongly correlated with a single notion of difficulty, and reveal examples that range from being highly sensitive to insensitive to the inductive biases of certain model architectures. Drawing from statistical genetics, we develop a simple method for fingerprinting model architectures using a few sensitive examples. These findings guide practitioners in maximizing the consistency of their scores (e.g. by choosing appropriate scoring methods, number of runs, and subsets of examples), and establishes comprehensive baselines for evaluating scores in the future.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑