用于持续学习的Soft-TransFormers
Soft-TransFormers for Continual Learning
AI总结：
Soft-TransFormers通过学习任务特定的乘法掩码实现持续学习，结合轻量级双提示机制，有效缓解灾难性遗忘并保持知识保留。
摘要：
受Well-initialized Lottery Ticket Hypothesis启发，我们引入Soft-Transformer（Soft-TF），一种参数高效的持续学习框架，利用冻结的预训练Transformer上的软实值子网络。不同于依赖手动设计的提示或适配器，Soft-TF学习应用于自注意力中键、查询、值和输出投影的任务特定乘法掩码。这些掩码在保持共享表示的同时实现平稳稳定的任务适应。结合轻量级双提示机制，Soft-TF保持了强大的知识保留并缓解了灾难性遗忘。在多个持续学习基准测试中，Soft-TF实现了最先进的性能，一致优于基于提示、适配器和LoRA风格的基线，同时仅需少量额外参数。
英文摘要：
Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers. The masks are initialized at one, so optimization starts exactly at the pre-trained solution, and mask-space gradient descent is intrinsically biased toward modulating the backbone's dominant pathways; we prove that, under standard convex-Lipschitz assumptions, both the convergence rate and the parameter drift of mask-only fine-tuning are controlled by the distance from the pre-trained weights to a task-optimal configuration. This bounded drift yields two properties. Since the backbone and per-task masks are never overwritten, forgetting is structurally eliminated. And since every task subnetwork stays near the shared pre-trained solution, a wrong mask still evaluates a near-generalist function, so task-inference errors are largely harmless and class-incremental accuracy is decoupled from task-inference reliability. As a plug-in, Soft-TF couples with L2P, DualPrompt, HiDe-Prompt, and NoRGa, selecting masks by task-key matching, an entropy-gradient criterion, or a learned task-identity classifier. Across class-incremental learning benchmarks -- Split-CIFAR100, Split-ImageNet-R, CUB-200, and 5-Datasets -- Soft-TF consistently outperforms prompt-based, adapter-based, and LoRA-style baselines at comparable trainable-parameter budgets, while keeping inference cost identical to the unmodified backbone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
