Switch-KD：视觉切换知识蒸馏用于视觉-语言模型
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
AI总结：
Switch-KD通过统一视觉-语言知识转移，解决多模态对齐问题，使小模型高效蒸馏大模型的多模态知识。
摘要：
视觉-语言模型（VLMs）在联合视觉-语言理解中表现出色，但其大规模部署面临资源受限场景的挑战。知识蒸馏（KD）提供了一种无需增加模型大小或数据需求的方法，提高部署效率。然而，应用KD到VLMs面临模态特定监督的挑战：尽管VLMs的多模态知识在语言空间融合，当前方法分别监督每个模态，未明确处理多模态对齐，导致不一致的多模态知识转移。为此，我们提出Switch-KD，一种视觉切换蒸馏框架，将视觉-语言知识转移统一到共享的文本概率空间。Switch-KD包含两个关键组件：（1）视觉切换蒸馏，将学生模型的视觉输出切换到教师模型的语言路径，构建跨模态概率参考，实现隐式视觉知识转移；（2）动态双向logits差异（DBiLD）损失，通过双向监督自适应对齐信息概率区域，同时保持教师和学生分布结构。指导Switch-KD，一个0.5B TinyLLaVA模型从3B教师模型中高效蒸馏出丰富的多模态知识，平均在10个多模态基准上提升3.6个百分点，无需任何架构修改。
英文摘要：
Vision-Language Models (VLMs) have shown remarkable capabilities in joint vision-language understanding, but their large scale poses significant challenges for deployment in resource-constrained scenarios. Knowledge Distillation (KD) offers a viable way to improve model capabilities without increasing model size or data requirements, making deployment more efficient. However, applying KD to VLMs is challenged by modality-specific supervision: although multimodal knowledge in VLMs is fused within the language space, current methods supervise each modality separately without explicitly addressing multimodal alignment, leading to inconsistent multimodal knowledge transfer. To address this, we propose Switch-KD, a visual-switch distillation framework that unifies vision-language knowledge transfer within a shared text-probability space. Switch-KD comprises two key components: (1) Visual-Switch Distillation, which switches the student's visual outputs into the teacher's language pathway to construct cross-modal probabilistic references for implicit visual knowledge transfer; and (2) Dynamic Bi-directional Logits Difference (DBiLD) loss, which adaptively aligns informative probability regions while preserving the distributional structures of teacher and student through bidirectional supervision. Guided by Switch-KD, a 0.5B TinyLLaVA effectively distills rich multimodal knowledge from its 3B teacher, yielding an average improvement of 3.6 points across 10 multimodal benchmarks without any architectural modification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
