正负动量：操纵随机梯度噪声以提高泛化能力
Positive-Negative Momentum: Manipulating Stochastic Gradient Noise to Improve Generalization
摘要：
众所周知，随机梯度噪声（SGN）作为深度学习的隐式正则化，对于深度网络的优化和泛化都至关重要。一些作品试图通过注入随机噪声来人工模拟SGN，以改善深度学习。然而，事实证明，注入的简单随机噪声不能像SGN那样工作，SGN是各向异性和参数依赖的。为了以低计算成本模拟SGN，并且不改变学习率或批量大小，我们提出了正负动量（PNM）方法，该方法是经典优化器中传统动量的强大替代方案。引入的PNM方法保持了两个近似独立的动量项。然后，我们可以通过调整动量差来显式地控制SGN的大小。我们从理论上证明了PNM的收敛保证和推广优势，随机梯度下降（SGD）。通过将PNM结合到两个传统的优化器，SGD与动量和亚当，我们的广泛的实验经验验证了显着的优势，基于PNM的变体相应的传统的基于动量的优化器。
英文摘要：
It is well-known that stochastic gradient noise (SGN) acts as implicit regularization for deep learning and is essentially important for both optimization and generalization of deep networks. Some works attempted to artificially simulate SGN by injecting random noise to improve deep learning. However, it turned out that the injected simple random noise cannot work as well as SGN, which is anisotropic and parameter-dependent. For simulating SGN at low computational costs and without changing the learning rate or batch size, we propose the Positive-Negative Momentum (PNM) approach that is a powerful alternative to conventional Momentum in classic optimizers. The introduced PNM method maintains two approximate independent momentum terms. Then, we can control the magnitude of SGN explicitly by adjusting the momentum difference. We theoretically prove the convergence guarantee and the generalization advantage of PNM over Stochastic Gradient Descent (SGD). By incorporating PNM into the two conventional optimizers, SGD with Momentum and Adam, our extensive experiments empirically verified the significant advantage of the PNM-based variants over the corresponding conventional Momentum-based optimizers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
