KOALA：一种带损失自适应的卡尔曼优化算法
KOALA: A Kalman Optimization Algorithm with Loss Adaptivity
AI总结：
针对神经网络训练中损失随机波动的随机优化问题，提出基于卡尔曼滤波的KOALA优化算法，可捕捉先进方法的梯度动力学，易实现且高效，性能比肩或优于现有SOTA优化算法。
摘要：
优化通常被建模为确定性问题，其解通过梯度下降等迭代过程求得。然而，在训练神经网络时，由于随机选取样本子集，损失函数会随（迭代）时间变化。这种随机性将优化问题转化为随机优化问题。我们提出将损失视为相对于某个参考最优值的带噪观测值。对损失的这种解读让我们能够采用卡尔曼滤波作为优化器，因为其递归公式正是为从带噪测量中估计未知参数而设计的。此外，我们证明用于未知参数演化的卡尔曼滤波动力学模型，能够捕捉Momentum（动量法）和Adam等先进方法的梯度动力学。我们将这种随机优化方法命名为KOALA，即Kalman Optimization Algorithm with Loss Adaptivity（带损失自适应的卡尔曼优化算法）的缩写。KOALA是一种易于实现、可扩展且高效的神经网络训练方法。我们提供了收敛性分析，并通过实验证明，在计算机视觉、语言建模等多个神经网络架构和机器学习任务中，该方法得到的参数估计效果与现有最先进优化算法相当，甚至更优。
英文摘要：
Optimization is often cast as a deterministic problem, where the solution is found through some iterative procedure such as gradient descent. However, when training neural networks the loss function changes over (iteration) time due to the randomized selection of a subset of the samples. This randomization turns the optimization problem into a stochastic one. We propose to consider the loss as a noisy observation with respect to some reference optimum. This interpretation of the loss allows us to adopt Kalman filtering as an optimizer, as its recursive formulation is designed to estimate unknown parameters from noisy measurements. Moreover, we show that the Kalman Filter dynamical model for the evolution of the unknown parameters can be used to capture the gradient dynamics of advanced methods such as Momentum and Adam. We call this stochastic optimization method KOALA, which is short for Kalman Optimization Algorithm with Loss Adaptivity. KOALA is an easy to implement, scalable, and efficient method to train neural networks. We provide convergence analysis and show experimentally that it yields parameter estimates that are on par with or better than existing state of the art optimization algorithms across several neural network architectures and machine learning tasks, such as computer vision and language modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
