arXiv 2604.224072026-07-27cs.LGcs.AI

Adam在持续学习中梯度修改的隐藏失败模式，以及自适应解耦动量路由作为修复

Hidden Failure Modes of Gradient Modification under Adam in Continual Learning, and Adaptive Decoupled Moment Routing as a Repair

Shanghai Jiao Tong University（上海交通大学）；Shanghai Maritime University（上海海洋大学）

Yuelin Hu, Zhenbo Yu, Zhengxue Cheng, Wei Liu, Li Song

备注：: The authors have identified substantial issues in the experimental design and evaluation that affect the validity of the main conclusions. Therefore, the authors have decided to withdraw this manuscript

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

研究揭示了Adam在持续学习中梯度修改的隐藏失败模式，提出自适应解耦动量路由作为修复方法，通过调整动量路径提升模型稳定性与性能。

摘要：

许多持续学习方法在上游修改梯度（如投影、惩罚重缩放、回放混合），而将Adam视为中性后端。我们展示了这种组合存在隐藏的失败模式。在高重叠、非适应性8域持续语言模型中，所有共享路由投影基线接近于原始遗忘（12.5-12.8 vs. 13.2）。一个0.5%的回放缓冲区是最强的共享替代方案，但仍达到11.6，而固定强度解耦低于原始值在14.1。只有自适应解耦路由保持稳定在9.4，比原始值提高3.8单位。在16域流中，其在最强共享路由投影基线上的增益增长到4.5-4.8单位。失败在干净基准上几乎不可见。我们通过Adam的第二动量路径解释这种效应：在测试范围内，投影导致旧方向有效学习率的1/(1-alpha)膨胀，与八个alpha值的测量在8%内一致。同样的冲突出现在惩罚方法、回放混合以及LoRA下的7B规模。我们的修复将修改的梯度仅路由到第一动量，同时保持幅度忠实的第二动量统计，具有重叠意识的自适应强度。这种简单改变是唯一经过测试的配置，能够一致避免崩溃，无论方法、优化器和规模。

英文摘要：

Many continual-learning methods modify gradients upstream (e.g., projection, penalty rescaling, replay mixing) while treating Adam as a neutral backend. We show this composition has a hidden failure mode. In a high-overlap, non-adaptive 8-domain continual LM, all shared-routing projection baselines collapse close to vanilla forgetting (12.5--12.8 vs. 13.2). A 0.5% replay buffer is the strongest shared alternative but still reaches 11.6, while fixed-strength decoupling falls below vanilla at 14.1. Only adaptive decoupled routing remains stable at 9.4, improving over vanilla by 3.8 units. On a 16-domain stream, its gain over the strongest shared-routing projection baseline grows to 4.5--4.8 units. The failure is largely invisible on clean benchmarks. We explain this effect through Adam's second-moment pathway: in the tested regime, projection induces a 1/(1-alpha) inflation of the old-direction effective learning rate, matching measurements within 8% across eight alpha values. The same conflict appears with penalty methods, replay mixing, and at 7B scale under LoRA. Our fix routes the modified gradient only to the first moment while preserving magnitude-faithful second-moment statistics, with overlap-aware adaptive strength. This simple change is the only tested configuration that consistently avoids collapse across methods, optimizers, and scale.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。