面向多模态任务持续学习的动态Transformer架构
Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks
AI总结：
针对现有持续学习方法多聚焦单模态、大型预训练Transformer微调成本高的问题，提出基于Transformer的TAM-CL框架，通过新增任务专属参数、知识蒸馏实现多模态任务持续学习，缓解灾难性遗忘，开销低且达SOTA性能。
摘要：
Transformer神经网络正日益取代先前的架构，应用于不同数据模态的广泛场景中。大型预训练Transformer神经网络的规模不断扩大，微调的计算需求持续增长，这给这类模型在需要边缘计算的应用中的广泛普及带来了重大挑战。为应对这一挑战，持续学习（CL）作为一种解决方案应运而生，它能够为自主学习智能体在顺序到达的任务间实现知识迁移。然而，当前的持续学习方法主要聚焦于纯视觉或纯语言的学习任务。我们提出了一种基于Transformer的持续学习框架，聚焦于同时涉及视觉与语言的学习任务，即视觉-语言（Vision-and-Language, VaL）任务。得益于Transformer在其他模态中的成功表现，我们的架构具备应用于多模态学习场景的潜力。在该框架中，我们通过向基础Transformer引入额外参数，使网络针对每个任务实现专门化，从而实现动态模型扩展，以顺序学习多个任务。我们还运用知识蒸馏技术，借助相关的过往经验更高效地学习当前任务。我们提出的方法——任务感知多模态持续学习（Task Attentive Multimodal Continual Learning, TAM-CL），能够在缓解灾难性遗忘问题的同时实现任务间的信息交换。值得注意的是，我们的方法具备可扩展性，仅产生极小的内存与时间开销。TAM-CL在具有挑战性的多模态任务上取得了当前最优（SOTA）性能。
英文摘要：
Transformer neural networks are increasingly replacing prior architectures in a wide range of applications in different data modalities. The increasing size and computational demands of fine-tuning large pre-trained transformer neural networks pose significant challenges for the widespread adoption of these models for applications that demand on-edge computing. To tackle this challenge, continual learning (CL) emerges as a solution by facilitating the transfer of knowledge across tasks that arrive sequentially for an autonomously learning agent. However, current CL methods mainly focus on learning tasks that are exclusively vision-based or language-based. We propose a transformer-based CL framework focusing on learning tasks that involve both vision and language, known as Vision-and-Language (VaL) tasks. Due to the success of transformers in other modalities, our architecture has the potential to be used in multimodal learning settings. In our framework, we benefit from introducing extra parameters to a base transformer to specialize the network for each task. As a result, we enable dynamic model expansion to learn several tasks in a sequence. We also use knowledge distillation to benefit from relevant past experiences to learn the current task more efficiently. Our proposed method, Task Attentive Multimodal Continual Learning (TAM-CL), allows for the exchange of information between tasks while mitigating the problem of catastrophic forgetting. Notably, our approach is scalable, incurring minimal memory and time overhead. TAM-CL achieves state-of-the-art (SOTA) performance on challenging multimodal tasks
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
