CompoDistill：多模态大语言模型中基于注意力的知识蒸馏用于组合推理
CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs
AI总结：
本文提出CompoDistill，通过显式对齐学生与教师模型的视觉注意力，提升多模态大语言模型的组合推理能力，同时保持视觉问答任务的性能。
摘要：
最近，高效的多模态大语言模型（MLLMs）因其高计算复杂度而受到关注，使其在实际应用中更具实用性。在此背景下，知识蒸馏（KD）方法作为一种有前途的替代方案，将丰富的视觉和语言知识从较大的模型（教师）转移到较小的模型（学生）。然而，我们发现现有的KD方法难以有效蒸馏教师MLLM的丰富视觉感知能力到学生，这一挑战在先前研究中被忽视。通过系统分析，我们发现学生和教师之间的视觉注意力不匹配是问题的主要原因。基于这一见解，我们提出了CompoDistill，一种新的KD框架，通过显式对齐学生和教师的视觉注意力，以增强学生视觉感知能力。我们的广泛实验表明，CompoDistill在需要视觉感知能力的组合推理任务中显著提高了性能，同时在视觉问答任务中的表现与现有研究相当。此外，CompoDistill在更先进的主干网络上也表现出有效性，突显了其通用性。
英文摘要：
Recently, efficient Multimodal Large Language Models (MLLMs) have gained significant attention as a solution to their high computational complexity, making them more practical for real-world applications. In this regard, the knowledge distillation (KD) approach has emerged as a promising alternative, which transfers the rich visual and linguistic knowledge from a larger model (teacher) to a smaller model (student). However, we observe that existing KD methods struggle to effectively distill the teacher MLLM's rich visual perception abilities to the student, a challenge that has been largely overlooked in previous studies. Through a systematic analysis, we identify visual attention misalignment between student and teacher as the main cause of this issue. Based on this insight, we propose CompoDistill, a novel KD framework that explicitly aligns the student's visual attention with that of the teacher to enhance the student's visual perception abilities. Our extensive experiments show that CompoDistill significantly improves performance on compositional reasoning tasks that require visual perception abilities while maintaining strong performance on visual question answering tasks, as done in existing studies. Furthermore, CompoDistill demonstrates effectiveness with a more advanced backbone, highlighting its generalizability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
