arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.22641cs.LGcs.AI
收藏

FastVLM:用于快速视觉-语言模型推理的自推测解码

FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference

Dept. of IEOR, IIT Bombay(工业工程与运营研究系,印度理工学院班加罗尔)

Divya Jyoti Bajpai, Manjesh Kumar Hanawal

AI总结:

FastVLM提出基于模仿学习的自推测解码框架,用轻量草稿模型自回归生成令牌并由完整模型非自回归验证,实现1.55-1.85倍加速且性能损失极小。

摘要:

视觉-语言模型(VLMs)在视觉理解和查询响应生成方面取得了显著进展,但由于自回归解码,常常面临高计算成本和推理延迟的挑战。在这项工作中,我们引入了一种基于模仿学习的自推测解码(SSD)框架,名为FastVLM,以解决这些局限性。我们的方法采用轻量级草稿模型以自回归方式生成令牌,而完整模型则非自回归地验证这些令牌。被接受的令牌无缝继续,而被拒绝的令牌由完整模型纠正,并用于指导草稿模型的改进。通过模仿网络,FastVLM通过整合来自完整模型架构的更深层次见解来增强草稿模型。同时,在训练草稿模型时,它保持了完整模型的性能完整性,实现了效率与准确性之间的平衡。与最终层相比,我们的方法将推理速度提高了1.55-1.85倍,而性能损失极小。

英文摘要:

Vision-language Models (VLMs) have made significant strides in visual understanding and query response generation, but often face challenges of high computational cost and inference latency due to autoregressive decoding. In this work, we introduce an imitation-learning-based Self-Speculative Decoding (SSD) framework, named FastVLM, to address these limitations. Our approach employs a lightweight draft model for token generation in an autoregressive manner, while a full model verifies these tokens non-autoregressively. Accepted tokens proceed seamlessly, while rejected tokens are corrected by the full model and used to guide the draft model's refinement. Through an imitation network, FastVLM enhances the draft model by integrating deeper level insights from the full model's architecture. Also, it maintains the performance integrity of the full model while training the draft model, achieving a balance between efficiency and accuracy. Our method speeds up the inference process by 1.55-1.85x as compared to the final layer with minimal loss in performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑