多模态大语言模型知识蒸馏的令牌级响应视觉注意力引导
Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation
AI总结：
研究多模态大语言模型知识蒸馏，挑战传统依赖，发现下游性能与响应视觉注意力相似度强相关。提出令牌级响应视觉注意力引导方法，通过自适应加权散度，有效引导学生模型，实验证明其性能优于基线。
摘要：
虽然知识蒸馏（KD）通过利用来自更大教师模型的监督被广泛用于训练轻量级模型，但仅依靠输出令牌分布已证明不足以压缩多模态大语言模型（MLLM）。由于输出令牌是模型关注视觉输入的副产品，先前的工作已经探索了明确地蒸馏注意力以提供直接监督信号。虽然很有前景，但要蒸馏哪些注意力信号的精确效用仍未得到充分探索。在这项工作中，我们通过揭示下游性能与对教师的响应视觉注意力相似度密切相关，但与提示条件注意力的相似度微不足道，从而挑战了对提示到视觉注意力的传统依赖。此外，我们观察到注意力分布在各个令牌之间表现出显著差异，表明统一的蒸馏目标是次优的。为此，我们引入了令牌级响应视觉注意力引导（TRAG），这是一种蒸馏目标，它1）将重点转移到响应视觉信号上，2）通过基于注意力熵自适应加权Kullback-Leibler散度来采用特定于令牌的目标，有效地引导学生反映教师精确视觉焦点。在多个基准上的广泛实验结果表明，TRAG明显优于先前的蒸馏基线。
英文摘要：
While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promising, the precise utility of which attention signals to distill remains under-explored. In this work, we challenge the conventional reliance on prompt-to-vision attention by revealing that downstream performance correlates strongly with response-to-vision attention similarity to the teacher, but negligibly with that of prompt-conditioned attention. Furthermore, we observe that attention distributions exhibit significant variance across individual tokens, indicating that a uniform distillation objective is suboptimal. To this end, we introduce Token-level Response-visual Attention Guidance (TRAG), a distillation objective that 1) shifts the focus to response-to-vision signals and 2) employs token-specific objectives by adaptively weighting the Kullback-Leibler divergence based on attention entropy, effectively guiding the student to mirror the teacher's precise visual focus. Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
