ParallelVLM: 无损视频-大语言模型加速与视觉对齐感知并行推测解码
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
AI总结：
ParallelVLM通过并行化阶段和无偏验证器引导剪枝策略，有效提升视频理解任务的解码效率，实现3.36倍的加速效果。
摘要：
尽管当前视频-大语言模型在视频理解任务中表现出色，但其自回归解码效率受限于大量视频标记数量。视觉标记剪枝可部分缓解这一瓶颈，但现有方法仍面临信息丢失和解码加速有限的问题。本文提出ParallelVLM，一种无需训练的草稿-验证推测解码框架，解决了长视频设置中草稿与目标模型之间相互等待和速度提升比例受限的问题。ParallelVLM包含两个并行阶段，最大化硬件利用率，并采用无偏验证器引导剪枝策略，通过消除注意力引导剪枝中的位置偏差，更好地对齐草稿与目标模型。大量实验表明，ParallelVLM能够有效扩展草稿窗口，接受长度高，且在LLaVA-Onevision-72B上加速视频理解基准3.36倍，在Qwen2.5-VL-32B上加速2.42倍，相比传统自回归解码。
英文摘要：
Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36$\times$ on LLaVA-Onevision-72B and 2.42$\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
