MASSV: 面向视觉语言模型推测解码的多模态适配与自数据蒸馏
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
AI总结：
针对小型语言模型作为视觉语言模型推测解码草稿模型时的视觉处理缺失与预测不匹配问题，提出两阶段MASSV方法，通过投影器连接视觉编码器与自蒸馏微调实现对齐，在多模型上最高提升30%接受长度、1.46倍端到端推理速度。
摘要：
推测解码通过让轻量级草稿模型生成多个令牌，再由更大的目标模型同时验证，可显著加速语言模型推理。然而，将该技术应用于视觉语言模型（VLM）存在两个核心挑战：可作为高效草稿模型的小型语言模型缺乏处理视觉输入的架构组件，且其令牌预测结果与考虑视觉上下文的VLM目标模型的预测结果不匹配。我们提出面向视觉语言模型推测解码的多模态适配与自数据蒸馏方法（MASSV），该方法通过两阶段流程将现有小型语言模型转化为有效的多模态草稿模型。MASSV首先通过一个轻量级可训练投影器，将目标VLM的视觉编码器连接到草稿模型，随后利用目标VLM生成的响应执行自蒸馏视觉指令微调，以对齐令牌预测。在Qwen2.5-VL和Gemma3模型系列上开展的综合实验表明，在视觉相关任务中，MASSV将接受长度提升最高达30%，并实现最高1.46倍的端到端推理加速。MASSV为加速现有及未来的VLM提供了一种可扩展、架构兼容的方法。
英文摘要：
Speculative decoding significantly accelerates language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. However, applying this technique to vision-language models (VLMs) presents two fundamental challenges: small language models that could serve as efficient drafters lack the architectural components to process visual inputs, and their token predictions fail to match those of VLM target models that consider visual context. We introduce Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (MASSV), which transforms existing small language models into effective multimodal drafters through a two-phase approach. MASSV first connects the target VLM's vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual instruction tuning using responses generated by the target VLM to align token predictions. Comprehensive experiments across the Qwen2.5-VL and Gemma3 model families demonstrate that MASSV increases accepted length by up to 30% and delivers end-to-end inference speedups of up to 1.46x on visually-grounded tasks. MASSV provides a scalable, architecture-compatible method for accelerating both current and future VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
