深度学习中隐式正则化的估计
Estimating Implicit Regularization in Deep Learning
AI总结：
本文提出通过梯度匹配方法估计深度学习中的隐式正则化，验证了其在处理复杂网络时的有效性，并展示了在Dropout等方法中的应用。
摘要：
深度学习系统已知具有隐式正则化（隐式偏置），倾向于选择简单解而非单纯最小化损失函数。在某些情况下，可以解析地推导隐式正则化，将其与等效惩罚连接起来。然而，现代深度学习系统复杂，包含训练过程和架构的修改（如早停、小批量、Dropout）等，其影响不总是直接可解释。尽管估计结果有助于理论家设计算法和从业者解释超参数选择，但此问题尚未得到直接关注。此外，隐式正则化是可处理的：正则化使权重更新偏离损失梯度，提供识别隐式偏置的信号。本文提供了一种梯度匹配方法，用于经验估计隐式正则化。该方法适用于已知正则化的网络，恢复流行的显式惩罚如ℓ1和ℓ2。它还复制了已知的隐式效应，如梯度下降中早停引起的二次权重惩罚，证明了其可用于测试隐式正则化的理论。关键的是，由于该方法是经验性的，它可以处理任意网络中的隐式正则化。我们通过表征Dropout在深度网络中的影响，展示了隐式ℓ2效应，表明该方法可用于理解具有隐式偏置的网络中的正则化。
英文摘要：
Deep learning systems are known to exhibit implicit regularization (alt. implicit bias), favoring simple solutions instead of merely minimizing the loss function. In some cases, we can analytically derive the implicit regularization -- connecting it to an equivalent penalty that augments the learning objective. However, modern deep learning systems are complex, carrying modifications to the training procedure and architecture (e.g. early stopping, minibatching, dropout) whose effects are not always directly interpretable. Although estimating the resulting implicit regularization could aid theorists in algorithm design and practitioners in interpreting their hyperparameter choices, this problem has received little direct attention. It is also tractable: regularization makes weight updates deviate from loss gradients, promising a signal for identifying implicit bias. Here we provide gradient matching methods that can be used to empirically estimate the implicit regularization. Our method works on networks with known regularization, recovering popular explicit penalties like $\ell_1$ and $\ell_2$. It also replicates known implicit effects, like the quadratic weight penalty induced by early stopping in gradient descent, demonstrating that it can be used to test theories of implicit regularization. Crucially, because our method is empirical, it can handle implicit regularization in arbitrary networks. We demonstrate this use by characterizing the effects of dropout in deep networks, showing implicit $\ell_2$ effects in this popular method. Our work shows that practitioners can use gradient matching to understand regularization in networks with implicit biases that are too complicated to derive analytically.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
