ZeroFlow：克服灾难性遗忘比你想象的更简单
ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Think
AI总结：
ZeroFlow 提出首个无梯度优化基准，证明仅靠前向传播即可缓解持续学习中的灾难性遗忘，并给出改进策略。
摘要：
反向传播为克服灾难性遗忘提供了一种通用配置。在持续学习和持续预训练中，SGD 和 Adam 等优化器通常用于权重更新。然而，由于黑盒 API、硬件限制或不可微系统，梯度信息在实际中并非总是可获取的，我们将这一挑战称为“梯度禁令”。为弥合这一差距，我们提出了 ZeroFlow，这是首个旨在评估无梯度优化算法以克服遗忘的基准。ZeroFlow 在一系列算法、遗忘场景和数据集上考察了一组基于前向传播的方法。我们的结果表明，仅靠前向传播就足以缓解遗忘。我们揭示了新的优化原理，凸显了基于前向传播的方法在缓解遗忘、管理任务冲突和降低内存需求方面的潜力。此外，我们提出了新的增强方法，仅使用前向传播即可进一步提高抗遗忘能力。这项工作为推进基于前向传播的持续学习方法的发展提供了必要的工具和见解。
英文摘要：
Backpropagation provides a generalized configuration for overcoming catastrophic forgetting. Optimizers such as SGD and Adam are commonly used for weight updates in continual learning and continual pre-training. However, access to gradient information is not always feasible in practice due to black-box APIs, hardware constraints, or non-differentiable systems, a challenge we refer to as the gradient bans. To bridge this gap, we introduce ZeroFlow, the first benchmark designed to evaluate gradient-free optimization algorithms for overcoming forgetting. ZeroFlow examines a suite of forward pass-based methods across various algorithms, forgetting scenarios, and datasets. Our results show that forward passes alone can be sufficient to mitigate forgetting. We uncover novel optimization principles that highlight the potential of forward pass-based methods in mitigating forgetting, managing task conflicts, and reducing memory demands. Additionally, we propose new enhancements that further improve forgetting resistance using only forward passes. This work provides essential tools and insights to advance the development of forward-pass-based methods for continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
