多模态的持续学习
Continual Learning for Multiple Modalities
AI总结：
针对现有持续学习方法仅适用单一模态的局限，提出一种多模态持续学习框架，通过自调节表示偏移巩固模态内知识并选择性整合跨模态信息，同时引入模态嵌入重对齐策略，有效缓解了多模态场景下的知识遗忘与干扰问题。
摘要：
持续学习旨在学习在连续时间步中观察到的任务知识，同时缓解对已学知识的遗忘。现有方法被设计为随时间学习单一模态（例如图像），这限制了它们在涉及多模态场景中的适用性。在本研究中，我们提出了一种适应多模态（图像、视频、音频、深度和文本）的新型持续学习框架。我们训练模型将各种模态与文本对齐，利用其丰富的语义信息。然而，这增加了遗忘已学知识的风险，并因任务间不同的输入特征而加剧。为缓解对模态先前知识的覆写，我们提出一个框架，在整合相关跨模态信息的同时巩固模态内知识。这是通过自调节学习表示的偏移，将新知识逐渐整合到跨模态保留的信息中来实现的。同时，它通过基于相互相关性选择性地整合先前遇到模态的知识，来缓解跨模态干扰。此外，我们引入了一种重新对齐模态嵌入的策略，有效解决模态间的偏置对齐。我们使用具有不同模态的多个数据集，在广泛的持续学习场景中评估了所提方法。大量实验表明，无论是否给定模态身份，我们的方法在这些场景中均优于现有方法。
英文摘要：
Continual learning aims to learn knowledge of tasks observed in sequential time steps while mitigating the forgetting of previously learned knowledge. Existing methods were designed to learn a single modality (e.g., image) over time, which limits their applicability in scenarios involving multiple modalities. In this work, we propose a novel continual learning framework that accommodates multiple modalities (image, video, audio, depth, and text). We train a model to align various modalities with text, leveraging its rich semantic information. However, this increases the risk of forgetting previously learned knowledge, exacerbated by the differing input traits across tasks. To alleviate the overwriting of previous knowledge of modalities, we propose a framework that consolidates intra-modal knowledge while incorporating relevant inter-modal information. This is achieved by self-regulating shifts in learned representations to gradually integrating novel knowledge into the information retained across modalities. Simultaneously, it mitigates inter-modal interference by selectively integrating knowledge from previously encountered modalities based on their mutual relevance. Furthermore, we introduce a strategy to re-align modality embeddings, effectively addressing biased alignment between modalities. We evaluate the proposed method in a wide range of continual learning scenarios using multiple datasets with different modalities. Extensive experiments demonstrate that ours outperforms existing methods in the scenarios, regardless of whether the identity of the modality is given.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
