一种改进的自适应PID优化器，具有增强的收敛性和稳定性，用于深度学习
An Improved Adaptive PID Optimizer with Enhanced Convergence and Stability for Deep Learning
- 1 Department of Computer Science \& Engineering, Indian Institute of Technology Indore, India.
- 3 National Remote Sensing Centre, Indian Space Research Organisation, India.
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本文提出了一种改进的自适应PID优化器IAdaPID-ADG，通过引入非递增有效学习率和基于梯度差的调制因子来解决AdaPID在收敛性和稳定性方面的不足，实验表明其在多个数据集上表现优异。
AI中文摘要：
优化在深度学习中至关重要。大多数优化器的基础方法是基于动量的随机梯度下降。然而，它有两个关键缺点。首先，它有噪声和变化的梯度，其次，它有超调现象。为了解决噪声梯度，提出了Adam，它仍然是最广泛使用的自适应优化器。为了解决超调现象，提出了一种基于控制理论的PID优化器。为了在单一框架内解决这些限制，最近提出了几种AdaPID的变体。尽管AdaPID表现良好，但它仍然继承了Adam的两个关键缺点，即收敛性和稳定性问题。在本文中，我们解决了这两个限制。为了修复收敛问题，我们独特地将使用非递增有效学习率的想法整合到AdaPID中（最初在AMSGrad中提出，是Adam的扩展）。为了修复稳定性问题，我们创新性地将基于梯度差的调制因子整合到AdaPID中（最初在DiffGrad中提出，是Adam的另一个扩展）。将这两种想法结合到AdaPID中，结果得到我们新的IAdaPID-ADG优化器。我们在多个数据集上评估了所提出的优化器，包括基准数据集（MNIST和CIFAR10）和实际数据集（IARC和AnnoCerv）。IAdaPID-ADG在所有竞争优化器中表现显著更好。此外，我们在MNIST数据集上进行了消融研究，以展示每个添加组件的贡献。
英文摘要：
Optimization is essential in deep learning. The foundational method upon which most optimizers are built is momentum-based stochastic gradient descent. However, it suffers from two key drawbacks. First, it has noisy and varying gradients, and second, it has an overshoot phenomenon. To address noisy gradients, Adam was proposed, which remains the most widely used adaptive optimizer. To address the overshoot phenomenon, a control-theory-based PID optimizer was proposed. To tackle both the limitations within a single framework, several variants of Adaptive PID (AdaPID) have recently been proposed. Although AdaPID performs well, it still inherits two critical drawbacks from Adam, namely convergence and stability issues. In this work, we address both these limitations. To fix the convergence issue, we uniquely integrate the idea of using a non-increasing effective learning rate into AdaPID (originally proposed in AMSGrad, an extension of Adam). To fix the stability issue, we innovatively integrate a gradient difference based modulation factor into AdaPID (originally proposed in DiffGrad, another extension of Adam). Combining both these ideas in AdaPID, results in our novel IAdaPID-ADG optimizer. We evaluate our proposed optimizer on multiple datasets, including benchmark datasets (MNIST and CIFAR10) and real-world datasets (IARC and AnnoCerv). The IAdaPID-ADG substantially outperforms all competing optimizers. Additionally, we perform an ablation study on the MNIST dataset to demonstrate the contribution of each added component.
