面向并行持续学习的稳定训练研究
Towards stable training of parallel continual learning
AI总结：
针对并行持续学习多任务同时训练引发的前后向传播不稳定问题，提出基于双块托普利茨矩阵正交约束和梯度正交分解的SPCL方法，有效提升训练稳定性与性能。
摘要：
并行持续学习（Parallel Continual Learning, PCL）任务研究多源输入下的持续学习训练方法，不同任务的数据随到达随学习。PCL训练效率高，非常适用于复杂多源数据系统，例如搭载多传感器的自动驾驶车辆。但在任意时刻都有多个任务需要同时训练，导致PCL存在严重的训练不稳定性，这种不稳定性体现在前向传播和反向传播过程中，会出现特征纠缠、梯度冲突的问题。本文提出了稳定并行持续学习（Stable Parallel Continual Learning, SPCL）这一新型方法，可同时提升PCL前向与反向传播的训练稳定性。在前向传播方面，我们对网络参数施加基于双块托普利茨（Doubly-block Toeplit, DBT）矩阵的正交性约束，以保证传播稳定一致。在反向传播方面，我们采用正交分解进行梯度管理，稳定反向传播过程并缓解跨任务的梯度冲突。通过保证正交性并最小化条件数来优化梯度，SPCL可有效稳定复杂优化任务中的梯度下降过程。实验结果表明，SPCL的性能优于现有最优方法，且具备更好的训练稳定性。
英文摘要：
Parallel Continual Learning (PCL) tasks investigate the training methods for continual learning with multi-source input, where data from different tasks are learned as they arrive. PCL offers high training efficiency and is well-suited for complex multi-source data systems, such as autonomous vehicles equipped with multiple sensors. However, at any time, multiple tasks need to be trained simultaneously, leading to severe training instability in PCL. This instability manifests during both forward and backward propagation, where features are entangled and gradients are conflict. This paper introduces Stable Parallel Continual Learning (SPCL), a novel approach that enhances the training stability of PCL for both forward and backward propagation. For the forward propagation, we apply Doubly-block Toeplit (DBT) Matrix based orthogonality constraints to network parameters to ensure stable and consistent propagation. For the backward propagation, we employ orthogonal decomposition for gradient management stabilizes backpropagation and mitigates gradient conflicts across tasks. By optimizing gradients by ensuring orthogonality and minimizing the condition number, SPCL effectively stabilizing the gradient descent in complex optimization tasks. Experimental results demonstrate that SPCL outperforms state-of-the-art methjods and achieve better training stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
