HAWAII：层次化视觉知识转移用于高效的视觉-语言模型
HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
AI总结：
HAWAII通过层次化视觉知识转移，高效整合多个视觉专家的知识，提升视觉-语言模型性能。
摘要：
改进视觉语言模型（VLMs）的视觉理解能力对于增强其在各种任务中的性能至关重要。尽管使用多个预训练的视觉专家已显示出巨大潜力，但在训练和推理过程中往往会带来显著的计算成本。为了解决这一挑战，我们提出了HAWAII，一个新颖的框架，该框架将多个视觉专家的知识蒸馏到一个视觉编码器中，使它能够继承多个专家的互补优势，同时保持最小的计算开销。为缓解不同教师之间的冲突并切换不同的教师特定知识，我们提出使用教师特定的低秩适应（LoRA）适配器，并配以相应的路由器。每个适配器都与特定的教师对齐，以避免蒸馏过程中的噪声指导。为了实现高效的知识蒸馏，我们提出了细粒度和粗粒度蒸馏。在细粒度层面，使用令牌重要性分数来强调每个教师中最信息量大的令牌。在粗粒度层面，我们汇总多个教师的知识，并使用一组通用知识LoRA适配器和路由器将其转移到学生模型中。在各种视觉-语言任务上的广泛实验显示，HAWAII相比流行的开源VLMs具有优势。代码可在https://github.com/yimuwangcs/wise-hawaii获得。
英文摘要：
Improving the visual understanding ability of vision-language models (VLMs) is crucial for enhancing their performance across various tasks. While using multiple pretrained visual experts has shown great promise, it often incurs significant computational costs during training and inference. To address this challenge, we propose HAWAII, a novel framework that distills knowledge from multiple visual experts into a single vision encoder, enabling it to inherit the complementary strengths of several experts with minimal computational overhead. To mitigate conflicts among different teachers and switch between different teacher-specific knowledge, instead of using a fixed set of adapters for multiple teachers, we propose to use teacher-specific Low-Rank Adaptation (LoRA) adapters with a corresponding router. Each adapter is aligned with a specific teacher, avoiding noisy guidance during distillation. To enable efficient knowledge distillation, we propose fine-grained and coarse-grained distillation. At the fine-grained level, token importance scores are employed to emphasize the most informative tokens from each teacher adaptively. At the coarse-grained level, we summarize the knowledge from multiple teachers and transfer it to the student using a set of general-knowledge LoRA adapters with a router. Extensive experiments on various vision-language tasks demonstrate the superiority of HAWAII compared to popular open-source VLMs. The code is available at https://github.com/yimuwangcs/wise-hawaii.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
