LLMs能够针对X模态推理在模态上持续演化
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
AI总结：
提出PathWeave框架，通过AnA适配器和MoE门控模块实现MLLMs的模态持续学习，无需联合模态预训练，在MCL基准上表现优异且减少98.73%训练负担。
摘要：
多模态大语言模型（MLLMs）因其在多模态理解方面的出色能力而备受关注。然而，现有方法严重依赖大量的模态特定预训练和联合模态微调，导致在扩展到新模态时产生巨大的计算负担。在本文中，我们提出了PathWeave，这是一个具有模态路径切换和扩展能力的灵活且可扩展的框架，使MLLMs能够持续在模态上演化以进行$\mathbb{X}$-模态推理。我们利用持续学习的概念，在预训练的MLLMs之上开发了一种增量训练策略，使其能够使用单模态数据扩展到新模态，而无需执行联合模态预训练。具体而言，引入了一种新颖的Adapter-in-Adapter (AnA)框架，其中单模态和跨模态适配器被无缝集成，以促进高效的模态对齐和协作。此外，在两种类型的适配器之间应用了基于MoE的门控模块，以进一步增强多模态交互。为了研究所提出的方法，我们建立了一个具有挑战性的基准，称为模态持续学习（MCL），它由来自五种不同模态的高质量问答数据组成：图像、视频、音频、深度和点云。大量实验证明了所提出的AnA框架在持续学习期间对学习可塑性和记忆稳定性的有效性。此外，PathWeave的性能与最先进的MLLMs相当，同时将参数训练负担减少了98.73%。我们的代码位于https://github.com/JiazuoYu/PathWeave
英文摘要：
Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding. However, existing methods rely heavily on extensive modal-specific pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modalities. In this paper, we propose PathWeave, a flexible and scalable framework with modal-Path sWitching and ExpAnsion abilities that enables MLLMs to continually EVolve on modalities for $\mathbb{X}$-modal reasoning. We leverage the concept of Continual Learning and develop an incremental training strategy atop pre-trained MLLMs, enabling their expansion to new modalities using uni-modal data, without executing joint-modal pretraining. In detail, a novel Adapter-in-Adapter (AnA) framework is introduced, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. Additionally, an MoE-based gating module is applied between two types of adapters to further enhance the multimodal interaction. To investigate the proposed method, we establish a challenging benchmark called Continual Learning of Modality (MCL), which consists of high-quality QA data from five distinct modalities: image, video, audio, depth and point cloud. Extensive experiments demonstrate the effectiveness of the proposed AnA framework on learning plasticity and memory stability during continual learning. Furthermore, PathWeave performs comparably to state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98.73%. Our code locates at https://github.com/JiazuoYu/PathWeave
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
