Align-KD：为移动视觉语言模型增强提取跨模态对齐知识
Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Model Enhancement
AI总结：
提出Align-KD方法，通过蒸馏教师模型浅层跨模态对齐知识，指导1.7B学生模型学习视觉-文本匹配，在6个基准上平均提升2.0分。
摘要：
视觉语言模型（VLM）为多模态任务带来了强大的理解和推理能力。同时，移动设备对强大人工智能的需求也日益增长，例如AI助手软件。一些工作试图将VLM迁移到边缘设备以扩展其应用范围。简化模型结构是一种常见方法，但随着模型缩小，性能与大小之间的权衡变得越来越困难。知识蒸馏（KD）可以帮助模型在不增加大小或数据量的情况下提升综合能力。然而，现有的大模型蒸馏技术大多只考虑单模态LLM的应用，或者仅使用教师为学生创建新的数据环境。这些方法都没有考虑VLM中最重要的跨模态对齐知识的蒸馏。我们提出了一种名为Align-KD的方法，引导学生模型学习发生在浅层的跨模态匹配。教师还帮助学生基于文本的关注点学习将视觉标记投影到文本嵌入空间。在Align-KD的指导下，1.7B的MobileVLM V2模型能够从7B教师模型中学习丰富的知识，且训练损失设计轻量，在两个训练子集上分别在6个基准上平均得分提升2.0。代码地址：https://github.com/fqhank/Align-KD。
英文摘要：
Vision-Language Models (VLMs) bring powerful understanding and reasoning capabilities to multimodal tasks. Meanwhile, the great need for capable aritificial intelligence on mobile devices also arises, such as the AI assistant software. Some efforts try to migrate VLMs to edge devices to expand their application scope. Simplifying the model structure is a common method, but as the model shrinks, the trade-off between performance and size becomes more and more difficult. Knowledge distillation (KD) can help models improve comprehensive capabilities without increasing size or data volume. However, most of the existing large model distillation techniques only consider applications on single-modal LLMs, or only use teachers to create new data environments for students. None of these methods take into account the distillation of the most important cross-modal alignment knowledge in VLMs. We propose a method called Align-KD to guide the student model to learn the cross-modal matching that occurs at the shallow layer. The teacher also helps student learn the projection of vision token into text embedding space based on the focus of text. Under the guidance of Align-KD, the 1.7B MobileVLM V2 model can learn rich knowledge from the 7B teacher model with light design of training loss, and achieve an average score improvement of 2.0 across 6 benchmarks under two training subsets respectively. Code is available at: https://github.com/fqhank/Align-KD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
