PILL：通过适配器专家与注意力门接入大语言模型
PILL: Plug Into LLM with Adapter Expert and Attention Gate
AI总结：
针对多模态任务中混合模态处理复杂的问题，提出带适配器专家与注意力门的PILL架构，通过模态独立处理、注意力自适应控及适配器优化，实现有竞争力的多模态融合性能。
摘要：
鉴于强大的大语言模型（LLMs）在遵循指令方面的卓越能力，社区中涌现出越来越多协助人类的智能助手。近期，视觉语言模型（VLMs）的研发取得了显著进展，拓展了LLMs的能力边界，使其能够执行更多样化的指令。但可以预见的是，未来模型很可能需要处理涉及语音、视频等更多模态的任务，这使得混合模态的复杂性处理成为一个尤为突出的挑战。为解决这一问题，我们提出了一种名为PILL（Plug Into LLM with adapter expert and attention gate，即带适配器专家与注意力门的LLM接入架构）的新型架构，以更好地解耦这些复杂模态并实现高效微调。我们引入了两个核心模块：其一，采用模态混合适配器专家（Mixture-of-Modality-Adapter-Expert）独立处理不同模态，在保留原始模型表达能力的同时，更好地适配下游任务；其二，引入模态注意力门控（Modality-Attention-Gating），可自适应控制模态令牌对整体表征的贡献程度。此外，我们还对适配器进行了改进，以增强其学习能力与表达能力。实验结果表明，与其他主流的模态融合方法相比，我们的方法展现出具有竞争力的性能。对本研究感兴趣的研究者可通过https://github.com/DsaltYfish/PILL免费获取代码与模型。
英文摘要：
Due to the remarkable capabilities of powerful Large Language Models (LLMs) in effectively following instructions, there has been a growing number of assistants in the community to assist humans. Recently, significant progress has been made in the development of Vision Language Models (VLMs), expanding the capabilities of LLMs and enabling them to execute more diverse instructions. However, it is foreseeable that models will likely need to handle tasks involving additional modalities such as speech, video, and others. This poses a particularly prominent challenge of dealing with the complexity of mixed modalities. To address this, we introduce a novel architecture called PILL: Plug Into LLM with adapter expert and attention gate to better decouple these complex modalities and leverage efficient fine-tuning. We introduce two modules: Firstly, utilizing Mixture-of-Modality-Adapter-Expert to independently handle different modalities, enabling better adaptation to downstream tasks while preserving the expressive capability of the original model. Secondly, by introducing Modality-Attention-Gating, which enables adaptive control of the contribution of modality tokens to the overall representation. In addition, we have made improvements to the Adapter to enhance its learning and expressive capabilities. Experimental results demonstrate that our approach exhibits competitive performance compared to other mainstream methods for modality fusion. For researchers interested in our work, we provide free access to the code and models at https://github.com/DsaltYfish/PILL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
