迈向视觉-语言模型蒸馏中的长窗口锚定
Towards Long-window Anchoring in Vision-Language Model Distillation
AI总结：
本文提出LAid方法，通过改进注意力机制和RoPE响应调节，提升视觉-语言模型的长上下文理解能力，使蒸馏模型的有效上下文窗口长度增加3.2倍。
摘要：
尽管大型视觉-语言模型（VLMs）在长上下文理解方面表现出色，但它们普遍的小分支在有限窗口大小下无法实现语言-摄影对齐。我们发现，知识蒸馏可以作为旋转位置嵌入（RoPE）的补充，以增强窗口大小（从大型模型中锚定）。基于这一见解，我们提出了LAid，它通过两个互补的组件直接旨在长距离注意力机制的转移：（1）一种渐进距离加权注意力匹配，训练过程中动态强调更长的位置差异；（2）一种可学习的RoPE响应增益调节，选择性地放大所需的位置敏感性。在多个模型家族上的广泛实验表明，LAid蒸馏模型的有效上下文窗口长度比基线小模型长多达3.2倍，同时在标准VL基准上保持或提高了性能。频谱分析也表明，LAid成功地保留了传统方法无法转移的关键低频注意力成分。我们的工作不仅为构建更高效的长上下文VLMs提供了实用技术，还为蒸馏过程中位置理解的产生和转移提供了理论见解。
英文摘要：
While large vision-language models (VLMs) demonstrate strong long-context understanding, their prevalent small branches fail on linguistics-photography alignment for a limited window size. We discover that knowledge distillation improves students' capability as a complement to Rotary Position Embeddings (RoPE) on window sizes (anchored from large models). Building on this insight, we propose LAid, which directly aims at the transfer of long-range attention mechanisms through two complementary components: (1) a progressive distance-weighted attention matching that dynamically emphasizes longer position differences during training, and (2) a learnable RoPE response gain modulation that selectively amplifies position sensitivity where needed. Extensive experiments across multiple model families demonstrate that LAid-distilled models achieve up to 3.2 times longer effective context windows compared to baseline small models, while maintaining or improving performance on standard VL benchmarks. Spectral analysis also suggests that LAid successfully preserves crucial low-frequency attention components that conventional methods fail to transfer. Our work not only provides practical techniques for building more efficient long-context VLMs but also offers theoretical insights into how positional understanding emerges and transfers during distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
