IAA：内部适配器架构赋能冻结大语言模型具备多模态能力
IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities
AI总结：
针对多模态大语言模型微调导致NLP能力下降的问题，提出内部适配器架构（IAA），在冻结语言模型不同深度集成多模态适配器，小规模数据即可提升性能，在视觉-语言基准测试中优于现有方法且不牺牲NLP能力。
摘要：
在多模态大语言模型（MLLMs）领域，常用方法通常在训练期间解冻语言模型以培养深度视觉理解能力。然而，使用视觉-语言数据对这类模型进行微调往往会导致其自然语言处理（NLP）能力下降。为避免这种性能退化，一个直接的解决方案是在开发多模态能力时冻结语言模型。但遗憾的是，先前的工作未能取得令人满意的结果。基于冻结语言模型的策略，我们进行了深入的结构探索，并引入了内部适配器架构（IAA）。具体而言，该架构在大语言模型的不同深度处集成了多个多模态适配器，以促进与固有文本导向的Transformer层的直接交互，从而使冻结的语言模型获得多模态能力。与以往需要大规模对齐数据的冻结语言模型方法不同，我们提出的架构能够在小规模数据集上实现更优性能。我们开展了广泛的实验以提升MLLM的通用多模态能力和视觉定位能力。我们的方法在各种视觉-语言基准测试中显著优于先前的最先进方法，同时不牺牲NLP任务的性能。代码和模型可在https://github.com/360CVGroup/Inner-Adaptor-Architecture获取。
英文摘要：
In the field of multimodal large language models (MLLMs), common methods typically involve unfreezing the language model during training to foster profound visual understanding. However, the fine-tuning of such models with vision-language data often leads to a diminution of their natural language processing (NLP) capabilities. To avoid this performance degradation, a straightforward solution is to freeze the language model while developing multimodal competencies. Unfortunately, previous works have not attained satisfactory outcomes. Building on the strategy of freezing the language model, we conduct thorough structural exploration and introduce the Inner-Adaptor Architecture (IAA). Specifically, the architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers, thereby enabling the frozen language model to acquire multimodal capabilities. Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets. We conduct extensive experiments to improve the general multimodal capabilities and visual grounding abilities of the MLLM. Our approach remarkably outperforms previous state-of-the-art methods across various vision-language benchmarks without sacrificing performance on NLP tasks. Code and models are available at https://github.com/360CVGroup/Inner-Adaptor-Architecture.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
