Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
University College London(伦敦大学学院);University of Cambridge(剑桥大学);MILA - Quebec AI institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Samsung - SAIT AI Lab Montreal(三星SAIT人工智能实验室蒙特利尔分部);Gatsby Computational Neuroscience Unit(加特兹计算神经科学中心);SWC - University College London(伦敦大学学院威尔科姆神经科学中心)
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup modeling different data sub-populations with a Gaussian-mixture model. We provide an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setting, which we prove to be exact in high dimension. Notably, our analysis reveals how different properties of sub-populations influence bias at different timescales, showing a shifting preference of the classifier during training. Applying our findings to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real datasets, including CIFAR10, MNIST, and CelebA.
Direction Matters: On the Implicit Bias of Stochastic Gradient Descent with Moderate Learning Rate
Johns Hopkins University(约翰霍普金斯大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
理解\emph{stochastic gradient descent}(SGD)的算法偏见是现代机器学习和深度学习理论的关键挑战之一。然而,大多数现有的工作集中在\emph{very small or even infinitesimal}学习率制度,并未能涵盖实际的情况下,学习率是\emph{moderate and annealing}。在本文中,我们作出了初步的尝试,以描述特定的正则化效果的SGD在中等学习率制度,通过研究其行为优化过参数化线性回归问题。在这种情况下,SGD和GD是已知的收敛到唯一的最小范数的解决方案,然而,与温和的和退火学习率,我们表明,他们表现出不同的\emph{directional bias}:SGD收敛沿大特征值方向的数据矩阵,而GD后的小特征值方向。此外,我们表明,这样的方向性偏差的问题时,采用早期停止,其中SGD输出接近最优,但GD输出是次优的。最后,我们的理论解释了几个民间艺术在实践中用于SGD超参数调整,如(1)线性缩放初始学习率与批量大小;(2)超越SGD高学习率,即使损失停止下降。
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);New York University(纽约大学);Facebook AI Research(Facebook人工智能研究院);Politecnico di Torino(都灵理工大学);Microsoft Research New England(微软研究院新英格兰分部)
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Texas A&M University(德克萨斯农工大学);Princeton University(普林斯顿大学);Massachusetts Institute of Technology(麻省理工学院);Center for Brains, Mind, and Machines(脑、心智与机器中心)
Bias-Aware Minimisation: Understanding and Mitigating Estimator Bias in Private SGD
Institute for AI in Medicine, Technical University of Munich(慕尼黑工业大学医学人工智能研究所);Konrad Zuse School for Excelllence in Reliable AI(康拉德·楚泽卓越可靠人工智能学院);V7 Labs(V7实验室);Imperial College London(帝国理工学院)
随机梯度下降(SGD)中的噪声为训练过参数化模型提供了至关重要的隐式正则化效应。以前的理论工作主要集中在球形高斯噪声,而经验研究表明,参数相关的噪声-由小批量或标签扰动引起的-比高斯噪声更有效。本文从理论上描述了这种现象的二次参数化模型介绍Vaskevicius等人。和Woodworth et el.我们表明,在一个过参数化的设置,SGD标签噪声恢复稀疏地面真理与任意初始化,而SGD高斯噪声或梯度下降过拟合到稠密的解决方案与大的规范。我们的分析表明,参数相关的噪声引入了一个偏向局部极小值较小的噪声方差,而球面高斯噪声没有。我们项目的代码是公开的。
Risk Bounds of Accelerated SGD for Overparameterized Linear Regression
University of California, Los Angeles(加州大学洛杉矶分校);Simons Institute, University of California, Berkeley(加州大学伯克利分校西蒙斯研究所);Indiana University Bloomington(印第安纳大学伯明顿分校)
Escaping Saddle Points with Bias-Variance Reduced Local Perturbed SGD for Communication Efficient Nonconvex Distributed Learning
NTT DATA Mathematical Systems Inc.(NTT数据数学系统公司);Graduate School of Information Science and Technology, The University of Tokyo(东京大学信息科学与技术研究生院);Center for Advanced Intelligence Project, RIKEN(理化学研究所先进智能项目中心)
在最近的集中式非凸分布式学习和联邦学习中,局部方法是减少通信时间的有前途的方法之一。然而,现有的工作主要集中在研究一阶最优性保证。另一方面,二阶最优性保证算法,即,逃逸鞍点的算法在非分布优化文献中已经被广泛研究。本文研究了一种新的局部算法--偏差方差降低局部扰动SGD(Bias-Variance Reduced Local Perturbed SGD,BVR-L-PSGD),它将已有的偏差方差降低梯度估计与参数扰动相结合,用于求解集中式非凸分布优化问题的二阶最优点。BVR-L-PSGD具有二阶最优性,但其通信复杂性与BVR-L-SGD的一阶最优性几乎相同。特别是当局部数据集的异质性小于局部损失的平滑性时,通信复杂性优于非局部方法。在极端情况下,当局部数据集的异质性为零时,通信复杂性度接近$\widetilde Θ(1)$。数值结果验证了我们的理论研究结果。
Increasing Missingness to Reduce Bias: Richardson-SGD with Missing Data
Sorbonne Université and Université Paris Cité, CNRS, Laboratoire de Probabilités, Statistique et Modélisation, LPSM(索邦大学和巴黎cité大学,CNRS,概率、统计与建模实验室,LPSM)
Reducing Bias in Deep Learning Optimization: The RSGDM Approach
Stevens Institute of Technology(史蒂文斯理工学院);University of California, Berkeley(加州大学伯克利分校);Chinese University of Hong Kong(香港中文大学);University of California, Irvine(加州大学欧文分校);Illinois Institute of Technology(伊利诺伊理工学院);Southern Methodist University(南卫理公会大学)
Neural networks trained with SGD learn distributions of increasing complexity
Université PSL(巴黎文理研究大学);CNRS(法国国家科学研究中心);Sorbonne Université(索邦大学);Université Paris-Diderot(巴黎第七大学);École Fédérale Polytechnique de Lausanne (EPFL)(洛桑联邦理工学院);The Abdus Salam International Centre for Theoretical Physics (ICTP)(阿卜杜勒·萨拉姆国际理论物理中心);International School of Advanced Studies (SISSA)(国际高等研究院)
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Alan Turing Institute(艾伦·图灵研究所);University of Oxford(牛津大学);Florida State University(佛罗里达州立大学);Rutgers Business School(罗格斯商学院);INRIA(法国国家信息与自动化研究所);École Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎文理研究大学)