缓解统一多模态模型持续学习中的模态内和模态间遗忘
Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models
AI总结：
本文提出MoDE，通过解耦模态以缓解统一多模态模型中的模态内和模态间遗忘问题。
摘要：
统一多模态生成模型（UMGMs）在单个自回归框架内统一了视觉理解和图像生成。然而，其持续学习新任务的能力严重受到灾难性遗忘的阻碍，这既发生在同一模态内（模态内遗忘），也发生在不同模态之间（模态间遗忘）。尽管先前持续学习（CL）工作已经研究了模态内遗忘，但模态间遗忘仍鲜有探索。在本文中，我们识别并实证了UMGMs中的这一现象，并提供了一个基于模态间梯度冲突的理论解释。为解决模态内和模态间遗忘，我们提出了模态解耦专家（MoDE），这是一种轻量且可扩展的架构，通过隔离模态特定的更新来缓解梯度冲突，并利用知识蒸馏来防止灾难性遗忘并保持预训练能力。与以往持续学习方法不同，后者仍保持模态耦合并受到模态梯度冲突的影响，MoDE明确解耦模态以防止干扰。在多样化的基准测试中，实验表明MoDE显著缓解了模态间和模态内遗忘，在统一多模态生成设置中优于先前的持续学习基线。代码将公开发布：https://github.com/Christina200/MoDE-official.git
英文摘要：
Unified Multimodal Generative Models (UMGMs) unify visual understanding and image generation within a single autoregressive framework. However, their ability to continually learn new tasks is severely hindered by catastrophic forgetting, both within a modality (intra-modal) and across modalities (inter-modal). While intra-modal forgetting has been studied in prior continual learning (CL) work, inter-modal forgetting remains largely unexplored. In this paper, we identify and empirically validate this phenomenon in UMGMs and provide a theoretical explanation rooted in gradient conflict between modalities. To address both intra- and inter-modal forgetting, we propose Modality-Decoupled Experts (MoDE), a lightweight and scalable architecture that isolates modality-specific updates to mitigate the gradient conflict and leverages knowledge distillation to prevent catastrophic forgetting and preserve pre-trained capabilities. Unlike previous CL methods that remain modality-coupled and suffer from modality gradient conflict, MoDE explicitly decouples modalities to prevent interference. Experiments across diverse benchmarks demonstrate that MoDE significantly mitigates both inter- and intra-modal forgetting, outperforming prior CL baselines in unified multimodal generation settings. Codes will be publicly available: https://github.com/Christina200/MoDE-official.git
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
