arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.18296cs.LGcond-mat.dis-nnstat.ML
收藏

Bias in Motion:SGD 训练中偏差动力学的理论洞察

Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training

University College London(伦敦大学学院);University of Cambridge(剑桥大学);MILA - Quebec AI institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Samsung - SAIT AI Lab Montreal(三星SAIT人工智能实验室蒙特利尔分部);Gatsby Computational Neuroscience Unit(加特兹计算神经科学中心);SWC - University College London(伦敦大学学院威尔科姆神经科学中心)

Anchit Jain, Rozhin Nobahari, Aristide Baratin, Stefano Sarao Mannelli

AI总结:

本文通过师生模型与高斯混合模型解析了线性分类器在 SGD 训练中的偏差瞬态动力学,揭示了子群体属性在不同时间尺度对偏差的影响,并在合成与真实数据集上验证了理论结果。

摘要:

机器学习系统常通过利用数据中的非期望特征而获得偏差,对不同子群体的影响程度各异。当前对偏差形成的理解多集中于学习的初始与最终阶段,对瞬态动力学存在认知空白。为填补此空白,本文探讨了在师生模型设置下偏差的演化,该设置使用高斯混合模型对不同的数据子群体进行建模。我们解析描述了此设定下线性分类器的随机梯度下降动力学,并证明其在高维下是精确的。值得注意的是,分析揭示了子群体的不同属性如何在不同时间尺度上影响偏差,表明分类器在训练期间偏好发生转变。将发现应用于公平性与鲁棒性,我们阐明了异构数据与虚假特征如何及何时生成并放大偏差。我们在更复杂场景中通过在合成与真实数据集(包括 CIFAR10、MNIST 和 CelebA)上训练更深网络来实证验证结果。

英文摘要:

Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup modeling different data sub-populations with a Gaussian-mixture model. We provide an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setting, which we prove to be exact in high dimension. Notably, our analysis reveals how different properties of sub-populations influence bias at different timescales, showing a shifting preference of the classifier during training. Applying our findings to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real datasets, including CIFAR10, MNIST, and CelebA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑