HIPPO：通过整体感知并行推测解码加速视频大语言模型推理
HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding
AI总结：
HIPPO通过整体感知并行推测解码框架，有效提升视频大语言模型推理速度，实现高达3.51倍的加速效果。
摘要：
推测解码（SD）已发展为一种有前景的方法，用于在不牺牲输出质量的情况下加速LLM推理。现有的针对视频LLM的SD方法主要集中在修剪冗余的视觉标记以减轻大规模视觉输入的计算负担。然而，现有方法并未达到与纯文本LLM相当的推理加速效果。我们通过广泛的实验发现，这种现象主要源于两个限制：（i）它们的修剪策略无法充分保留视觉语义标记，从而降低草稿质量和接受率；（ii）即使进行激进的修剪（例如，移除90%的视觉标记），剩余的草稿模型推理成本也限制了整体加速效果。为了解决这些限制，我们提出了HIPPO，一种通用的整体感知并行推测解码框架。具体来说，HIPPO提出了（i）一种语义感知的标记保留方法，该方法融合了全局注意力分数与局部视觉语义以在高修剪比率下保留语义信息；（ii）一种视频并行SD算法，该算法将草稿生成和目标验证阶段解耦并重叠。在四个视频LLM上六个基准测试中的实验表明，HIPPO的有效性，其速度提升最高可达3.51倍相比于原始自回归解码。
英文摘要：
Speculative decoding (SD) has emerged as a promising approach to accelerate LLM inference without sacrificing output quality. Existing SD methods tailored for video-LLMs primarily focus on pruning redundant visual tokens to mitigate the computational burden of massive visual inputs. However, existing methods do not achieve inference acceleration comparable to text-only LLMs. We observe from extensive experiments that this phenomenon mainly stems from two limitations: (i) their pruning strategies inadequately preserve visual semantic tokens, degrading draft quality and acceptance rates; (ii) even with aggressive pruning (e.g., 90% visual tokens removed), the draft model's remaining inference cost limits overall speedup. To address these limitations, we propose HIPPO, a general holistic-aware parallel speculative decoding framework. Specifically, HIPPO proposes (i) a semantic-aware token preservation method, which fuses global attention scores with local visual semantics to retain semantic information at high pruning ratios; (ii) a video parallel SD algorithm that decouples and overlaps draft generation and target verification phases. Experiments on four video-LLMs across six benchmarks demonstrate HIPPO's effectiveness, yielding up to 3.51x speedup compared to vanilla auto-regressive decoding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
