从架构视角重新思考持续学习中的稳定性-可塑性权衡
Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective
AI总结：
本文从架构层面揭示深度网络可塑性更强、宽度网络稳定性更强的规律，并提出即插即用的 Dual-Arch 框架，通过两个专用轻量网络分别负责可塑性与稳定性，在提升持续学习性能的同时最多减少 87% 参数。
摘要：
持续学习（Continual Learning, CL）的研究旨在赋予神经网络增量学习与适应的能力。这一追求的核心在于解决稳定性-可塑性困境，即在两个相互冲突的目标之间取得平衡：保留先前学到的知识和获取新知识。尽管大量持续学习方法试图实现这一权衡，但它们往往忽视了网络架构对稳定性和可塑性的影响，将权衡限制在参数层面。本文从架构层面深入探讨稳定性与可塑性之间的冲突。我们揭示，在相同参数约束下，更深的网络表现出更好的可塑性，而更宽的网络则以更强的稳定性为特征。为了解决这一架构层面的困境，我们引入了一个名为 Dual-Arch 的新框架，它可作为持续学习的即插即用组件。该框架利用两个不同且相互独立的网络的互补优势：一个专注于可塑性，另一个专注于稳定性。每个网络都采用专门且轻量级的架构设计，以适配其各自的目标。大量实验表明，Dual-Arch 提升了现有持续学习方法的性能，同时在参数方面紧凑性最高可达 87%。代码：https://github.com/byyx666/Dual-Arch。
英文摘要：
The quest for Continual Learning (CL) seeks to empower neural networks with the ability to learn and adapt incrementally. Central to this pursuit is addressing the stability-plasticity dilemma, which involves striking a balance between two conflicting objectives: preserving previously learned knowledge and acquiring new knowledge. While numerous CL methods aim to achieve this trade-off, they often overlook the impact of network architecture on stability and plasticity, restricting the trade-off to the parameter level. In this paper, we delve into the conflict between stability and plasticity at the architectural level. We reveal that under an equal parameter constraint, deeper networks exhibit better plasticity, while wider networks are characterized by superior stability. To address this architectural-level dilemma, we introduce a novel framework denoted Dual-Arch, which serves as a plug-in component for CL. This framework leverages the complementary strengths of two distinct and independent networks: one dedicated to plasticity and the other to stability. Each network is designed with a specialized and lightweight architecture, tailored to its respective objective. Extensive experiments demonstrate that Dual-Arch enhances the performance of existing CL methods while being up to 87% more compact in terms of parameters. Code: https://github.com/byyx666/Dual-Arch.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
