SpecFLASH: 一种基于潜在引导的半自回归推测解码框架，用于高效多模态生成
SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation
AI总结：
SpecFLASH是一种针对多模态生成的高效推测解码框架，通过潜在引导的token压缩和半自回归解码方案，显著提升视觉任务的解码速度。
摘要：
大型语言模型和大型多模态模型（LLMs和LMMs）在生成性能方面表现优异，但解码过程缓慢，尤其是在处理视觉输入时，问题更加突出，因为视觉序列通常包含更多token，信息密度较低。推测解码通过让紧凑的草稿模型提出候选token，由更大的目标模型选择性接受，从而在不降低质量的情况下加速LLM推理。然而，现有的多模态推测解码方法大多忽略了视觉表示的结构特征，通常依赖于文本-only的草稿模型。在本文中，我们引入了SpecFLASH，一种针对LMMs的推测解码框架，其在设计草稿模型时明确利用了多模态结构。我们首先通过一个轻量级、潜在引导的token压缩模块来减轻视觉token序列中的冗余，该模块在压缩视觉特征的同时保留语义，然后利用视觉实体的共现和局部相关性，通过半自回归解码方案在单次前向传递中预测多个token。广泛的实验表明，SpecFLASH在视频描述和视觉指令微调任务上分别比原始LMM实现了高达2.68倍和2.55倍的加速。我们的代码已在此处提供：https://github.com/ZihuaEvan/FlashSD/
英文摘要：
Large language models and large multimodal models (LLMs and LMMs) deliver strong generative performance but suffer from slow decoding, a problem that becomes more severe when handling visual inputs, whose sequences typically contain many more tokens with lower information density than text. Speculative decoding accelerates LLM inference by letting a compact draft model propose candidate tokens that are selectively accepted by a larger target model, achieving speed-up without degrading quality. However, existing multimodal speculative decoding approaches largely ignore the structural characteristics of visual representations and usually rely on text-only draft models. In this paper, we introduce SpecFLASH, a speculative decoding framework tailored to LMMs that explicitly exploits multimodal structure when designing the draft model. We first mitigate redundancy in visual token sequences with a lightweight, latent-guided token compression module that compacts visual features while preserving semantics, and then leverage the co-occurrence and local correlations of visual entities via a semi-autoregressive decoding scheme that predicts multiple tokens in a single forward pass. Extensive experiments demonstrate that SpecFLASH consistently surpasses prior speculative decoding baselines, achieving up to $2.68\times$ speed-up on video captioning and $2.55\times$ on visual instruction tuning, relative to the original LMM. Our code is available here: https://github.com/ZihuaEvan/FlashSD/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
