AdaSwarm：用群体智能增强深度学习中的梯度优化器
AdaSwarm: Augmenting Gradient-Based optimizers in Deep Learning with Swarm Intelligence
AI总结：
AdaSwarm是一种无梯度优化器，通过提出的EMPSO近似梯度，性能媲美Adam，能处理多种损失函数，包括MAE。
摘要：
本文介绍了AdaSwarm，一种新颖的无梯度优化器，其性能与神经网络中采用的Adam优化器相当甚至更优。为了支持我们提出的AdaSwarm，我们提出了一种新颖的指数加权动量粒子群优化器（EMPSO）。AdaSwarm解决优化问题的能力归功于其进行良好梯度近似的能力。我们证明，任何函数（无论可微与否）的梯度都可以利用EMPSO的参数来近似。这是一种模拟梯度下降的新技术，处于数值方法与群体智能的交叉领域。我们还提供了所产生梯度近似的数学证明。AdaSwarm与几种最先进的（SOTA）优化器竞争激烈。我们还表明，AdaSwarm能够在反向传播过程中处理各种损失函数，包括最大绝对误差（MAE）。
英文摘要：
This paper introduces AdaSwarm, a novel gradient-free optimizer which has similar or even better performance than the Adam optimizer adopted in neural networks. In order to support our proposed AdaSwarm, a novel Exponentially weighted Momentum Particle Swarm Optimizer (EMPSO), is proposed. The ability of AdaSwarm to tackle optimization problems is attributed to its capability to perform good gradient approximations. We show that, the gradient of any function, differentiable or not, can be approximated by using the parameters of EMPSO. This is a novel technique to simulate GD which lies at the boundary between numerical methods and swarm intelligence. Mathematical proofs of the gradient approximation produced are also provided. AdaSwarm competes closely with several state-of-the-art (SOTA) optimizers. We also show that AdaSwarm is able to handle a variety of loss functions during backpropagation, including the maximum absolute error (MAE).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
