MoExtend：为模态与任务扩展调优新专家
MoExtend: Tuning New Experts for Modality and Task Extension
AI总结：
MoExtend通过向预训练MoE模型无缝集成新专家，在不调优原有模型的情况下实现模态与任务扩展，从而避免灾难性遗忘并高效增强LLM多模态能力。
摘要：
大语言模型（LLMs）在各种任务中表现出色，但主要基于文本数据进行训练，限制了其应用范围。扩展LLM能力以涵盖视觉-语言理解至关重要，然而从头在多模态数据上训练它们既困难又昂贵。现有的指令调优方法，例如LLAVA，通常通过全量微调LLMs来连接预训练的CLIP视觉编码器和LLMs，以弥合模态差距。然而，全量微调受到灾难性遗忘的困扰，即遗忘先前知识，并且在任务和模态日益增多的时代训练成本高昂。为解决这一问题，我们提出了MoExtend，这是一个旨在简化Mixture-of-Experts（MoE）模型模态适应与扩展的有效框架。MoExtend无缝地将新专家集成到预训练的MoE模型中，赋予它们新知识，而无需调优预训练模型（如MoE和视觉编码器）。这种方法能够快速适应和扩展到新的模态数据或任务，有效解决了在LLMs中容纳新模态的挑战。此外，MoExtend避免调优预训练模型，从而降低了灾难性遗忘的风险。实验结果证明了MoExtend在增强LLMs多模态能力方面的有效性和效率，为多模态AI研究的进展做出了贡献。代码：https://github.com/zhongshsh/MoExtend。
英文摘要：
Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research. Code: https://github.com/zhongshsh/MoExtend.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
