仅查询注意力的持续学习
Continual Learning with Query-Only Attention
AI总结：
提出仅查询注意力机制，通过丢弃键和值保留transformer归纳偏置，显著缓解持续学习中的可塑性丧失与灾难性遗忘，并证明全注意力非捕获元学习优势所必需。
摘要：
持续学习涉及从无数据点重复的数据流中学习，由于任务间的分布偏移，这一场景本质上十分复杂。我们提出了一种仅查询注意力机制，它丢弃了键和值，但保留了transformer架构的核心归纳偏置。在持续学习场景中，这种简化机制显著缓解了可塑性丧失和灾难性遗忘，优于选择性重新初始化等基线方法。我们建立了仅查询注意力、全transformer注意力与模型无关元学习之间的概念联系，将它们构建为元学习的实例。我们进一步提供了直觉解释，说明为何基于查询的模型和注意力网络有助于在持续学习环境中保持可塑性。最后，通过初步的Hessian谱分析，我们观察到在跨任务中保持更高曲率秩的模型往往能保留可塑性。我们的研究结果表明，全注意力可能并非在持续学习中捕获元学习优势所必需。
英文摘要：
Continual learning involves learning from a stream of data without repetition of data points, a scenario that is inherently complex due to distributional shift across tasks. We propose a query-only attention mechanism that discards keys and values, yet preserves the core inductive bias of transformer architectures. In continual learning scenarios, this simplified mechanism significantly mitigates both loss of plasticity and catastrophic forgetting, outperforming baselines such as selective re-initialization. We establish a conceptual link between query-only attention, full transformer attention, and model agnostic meta-learning, framing them as instances of meta-learning. We further provide intuition for why query-based models and attention networks help preserve plasticity in continual settings. Finally, through preliminary Hessian spectrum analysis, we observe that models maintaining higher curvature rank across tasks tend to retain plasticity. Our findings suggest that full attention may not be essential for capturing the benefits of meta-learning in continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
