CSER: Communication-efficient SGD with Error Reset

随机重置缓解标签噪声导致的 SGD 潜在梯度偏差

Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise

本文针对噪声标签使 SGD 产生潜在梯度偏差并损害泛化的问题，引入统计物理中的随机重置机制，理论界定其有效条件并实验验证可显著提升泛化且易于兼容现有方法。

泊松重置的岭回归：谱正则化的更新视角

Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization

通过非平衡统计物理中的随机重置与统计学习中的岭正则化建立联系，证明线性梯度流下以速率r重置到原点产生的稳态均值即为岭估计，并推广到一般更新重置律以生成替代谱滤波器。

Experiments with Rich Regime Training for Deep Learning

In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the…

Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent

Stochastic gradient descent (SGD) with constant momentum and its variants such as Adam are the optimization algorithms of choice for training deep neural networks (DNNs). Since DNN training is incredibly computationally expensive, there is great interest in speeding up the convergence. Nesterov accelerated gradient (NAG) improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst. In this paper, we propose Sc…

Manipulating SGD with Data Ordering Attacks

Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even …

SGD过程张量成像：通过可区分性的回流测量非马尔可夫记忆

Process-Tensor Tomography of SGD: Measuring Non-Markovian Memory via Back-Flow of Distinguishability

本文提出通过可区分性回流测量SGD训练中的非马尔可夫记忆，提供了一个模型无关的见证方法，展示了优化器状态记忆对训练过程的影响。