MCD：大型视觉-语言模型中多模态上下文学习的大语言模型因果蒸馏

中文摘要 AI 辅助

大型视觉-语言模型（LVLMs）展现出强大的多模态上下文学习（ICL）能力，然而随着模型规模的减小，这种能力会大幅下降。知识蒸馏为弥合这一差距提供了一种自然途径，但现有方法主要直接对齐输出分布或隐藏表示。这种对齐方式教会学生模型教师模型预测了什么，却未揭示复杂上下文中哪些证据因果性地支持该预测。因此，学生模型可能模仿教师模型的答案，却继续依赖语言先验、提示结构或其他虚假线索。为解决这一局限，我们提出了多模态因果蒸馏（MCD），这是一种蒸馏框架，用于迁移强教师模型在ICL期间如何使用多模态证据。MCD采用保持结构的令牌干预来识别和验证因果证据，然后迁移教师模型在保留或移除该证据时的响应方式。这一设计将蒸馏与模型在多模态ICL期间使用上下文证据的因果模式联系起来。在三个LVLM家族和七个基准上的实验表明，MCD平均将学生模型性能提升了7.23个百分点，比普通蒸馏高出4.68个百分点，进一步分析证实了这些提升的泛化性。

英文摘要

Large vision-language models (LVLMs) exhibit strong multimodal in-context learning (ICL) capabilities, yet this ability degrades substantially as model size decreases. Knowledge distillation offers a natural way to bridge this gap, but existing methods primarily align output distributions or hidden representations directly. Such alignment teaches the student what the teacher predicts without revealing which evidence in the complex context causally supports that prediction. Consequently, a student can imitate the teacher's answer while continuing to rely on language priors, prompt structure, or other spurious cues. To address this limitation, we introduce Multimodal Causal Distillation (MCD), a distillation framework that transfers how a strong teacher uses multimodal evidence during ICL. MCD uses structure-preserving token interventions to identify and verify causal evidence, then transfers how the teacher responds when that evidence is retained or removed. This design connects distillation to the causal patterns by which the model uses contextual evidence during multimodal ICL. Experiments across three LVLM families and seven benchmarks show that MCD improves student performance by 7.23 points on average and outperforms vanilla distillation by 4.68 points, while further analyses confirm the generalizability of these gains.

发表机构

• Brown University（布朗大学）
• University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）
• Hanyang University（汉阳大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

17 pages, 8 tables, 5 figures