Continual-NExT: 一种统一的理解和生成持续学习框架
Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework
AI总结：
本文提出Continual-NExT框架，通过MAGE方法提升双到双多模态大语言模型的持续学习能力，实现跨模态知识转移和减少遗忘。
摘要：
双到双多模态大语言模型（Dual-to-Dual MLLMs）是指能够通过文本和图像模态实现统一多模态理解和生成的多模态大语言模型。尽管表现出强大的即时学习和泛化能力，双到双多模态大语言模型在终身进化方面仍然存在缺陷，显著影响其在动态现实场景中的持续适应能力。其中一个挑战是学习新任务不可避免地会破坏已学习的知识。除了传统灾难性遗忘之外，双到双多模态大语言模型还面临其他挑战，包括幻觉、指令不遵循以及跨模态知识转移的失败。然而，至今尚未建立标准化的双到双多模态大语言模型持续学习框架，这些挑战仍未被探索。因此，在本文中，我们建立了Continual-NExT，一种为双到双多模态大语言模型设计的持续学习框架，其评估指标经过精心设计。为了提高双到双多模态大语言模型的持续学习能力，我们提出了一种高效的MAGE（混合和聚合通用LoRA和专家LoRA）方法，以进一步促进跨模态的知识转移并缓解遗忘。大量实验表明，MAGE在其他持续学习方法中表现更优，并实现了最先进的性能。
英文摘要：
Dual-to-Dual MLLMs refer to Multimodal Large Language Models, which can enable unified multimodal comprehension and generation through text and image modalities. Although exhibiting strong instantaneous learning and generalization capabilities, Dual-to-Dual MLLMs still remain deficient in lifelong evolution, significantly affecting continual adaptation to dynamic real-world scenarios. One of the challenges is that learning new tasks inevitably destroys the learned knowledge. Beyond traditional catastrophic forgetting, Dual-to-Dual MLLMs face other challenges, including hallucination, instruction unfollowing, and failures in cross-modal knowledge transfer. However, no standardized continual learning framework for Dual-to-Dual MLLMs has been established yet, leaving these challenges unexplored. Thus, in this paper, we establish Continual-NExT, a continual learning framework for Dual-to-Dual MLLMs with deliberately-architected evaluation metrics. To improve the continual learning capability of Dual-to-Dual MLLMs, we propose an efficient MAGE (Mixture and Aggregation of General LoRA and Expert LoRA) method to further facilitate knowledge transfer across modalities and mitigate forgetting. Extensive experiments demonstrate that MAGE outperforms other continual learning methods and achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
