利用模态感知特征蒸馏增强视觉问答中的持续学习
Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation
AI总结：
针对视觉问答持续学习中多模态输入演化速率不同导致遗忘的问题，提出模态感知特征蒸馏（MAFED）方法，有效提升了多模态持续学习性能并防止遗忘。
摘要：
持续学习专注于在一系列任务上增量训练模型，旨在学习新任务的同时最小化在先前任务上的性能下降。现有处于持续学习和视觉问答（VQA）交叉领域的方法，未研究输入的多模态特性如何影响模型的学习动态。在本文中，我们证明了每种模态在任务连续体中以不同速率演化，且这种行为既出现在已有的纯编码器模型中，也出现在开发视觉与语言（VL）模型的现代方案中。受此观察启发，我们提出了一种模态感知特征蒸馏（MAFED）方法，在三种多模态持续学习设置中，该方法在不同规模的模型上均优于现有基线。此外，我们提供了消融实验，表明模态感知蒸馏是对经验回放的有效补充。总体而言，我们的结果强调了解决模态特定动态以防止多模态持续学习中遗忘的重要性。
英文摘要：
Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
