无遗忘的路由
Routing without Forgetting
AI总结：
无遗忘的路由通过在transformer中引入基于能量的关联检索层，实现了在在线持续学习中动态选择表示子空间，从而在多个基准测试中超越了现有基于提示的方法。
摘要：
在transformer中进行持续学习通常通过参数高效适应来解决：提示、适配器或LoRA模块会针对每个任务进行专门化，而主干网络保持冻结。尽管在受控的多轮设置中效果显著，这些方法依赖于渐进的梯度基于专门化，而在在线持续学习（OCL）中表现不佳，因为数据以非平稳流的形式出现，每个样本可能只被观察一次。我们重新将transformer中的持续学习视为一个路由问题：在严格的在线约束下，模型必须在没有显式任务标识符或重复优化的情况下，为每个输入动态选择适当的表示子空间。因此，我们引入了无遗忘的路由（RwF），一种在transformer架构中加入了基于能量的关联检索层的架构，灵感来自现代Hopfield网络。而不是存储或合并任务特定的提示，RwF通过在每个层上对transformer标记嵌入进行单步关联检索来生成动态提示。检索对应于严格凸自由能功能的闭式最小化，使在每次前向传递中都能根据输入条件进行路由，而无需迭代梯度细化。在具有挑战性的类增量基准上，RwF优于现有基于提示的方法。在Split-ImageNet-R和Split-ImageNet-S上，RwF在少量样本学习情况下也大幅优于先前的基于提示的方法。这些结果表明，在transformer主干中直接嵌入基于嵌入能量的关联路由提供了OCL的原理和有效基础。
英文摘要：
Continual learning in transformers is commonly addressed through parameter-efficient adaptation: prompts, adapters, or LoRA modules are specialized per task while the backbone remains frozen. Although effective in controlled multi-epoch settings, these approaches rely on gradual gradient-based specialization and struggle in Online Continual Learning (OCL), where data arrive as a non-stationary stream and each sample may be observed only once. We recast continual learning in transformers as a routing problem: under strict online constraints, the model must dynamically select the appropriate representational subspace for each input without explicit task identifiers or repeated optimization. We thus introduce Routing without Forgetting (RwF), a transformer architecture augmented with energy-based associative retrieval layers inspired by Modern Hopfield Networks. Instead of storing or merging task-specific prompts, RwF generates dynamic prompts through single-step associative retrieval over the transformer token embeddings at each layer. Retrieval corresponds to the closed-form minimization of a strictly convex free-energy functional, enabling input-conditioned routing within each forward pass, independently of iterative gradient refinement. Across challenging class-incremental benchmarks, RwF improves over existing prompt-based methods. On Split-ImageNet-R and Split-ImageNet-S, RwF outperforms prior prompt-based approaches by a large margin, even in few-shot learning regimes. These results indicate that embedding energy-based associative routing directly within the transformer backbone provides a principled and effective foundation for OCL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
