通过蒸馏和强化学习生长多头Twig以加速大视觉-语言模型
Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Models
AI总结：
本文提出TwigVLM，通过在基础VLM早期层上生长轻量模块Twig，结合 Twig 引导的 token 剪枝和自推测解码策略，实现更高的准确性和速度。实验表明， TwigVLM 在剪枝88.9%的视觉token后仍保持96%的原始性能，并在生成长响应时达到154%的速度提升。
摘要：
大视觉-语言模型（VLMs）在开放世界多模态理解中展现了显著能力，但其高计算开销对实际部署构成挑战。一些近期工作提出了通过剪枝冗余视觉token来加速VLMs的方法，这些方法基于VLM早期层的注意力图进行剪枝。尽管这些token剪枝方法取得成功，但仍存在两个主要不足：（i）由于早期层注意力信号不敏感导致显著精度下降；（ii）在生成长响应（如30个token）时速度提升有限。为了解决上述限制，我们提出TwigVLM——一种简单且通用的架构，通过在基础VLM的早期层上生长一个轻量模块，称为twig。与大多数现有VLM加速方法仅基于视觉token剪枝不同，我们的TwigVLM 通过采用twig引导的token剪枝（TTP）策略实现了更好的精度保持，并通过自推测解码（SSD）策略实现了更高的生成速度。以LLaVA-1.5-7B为基VLM，实验结果表明，TwigVLM 在剪枝88.9%的视觉token后仍保持96%的原始性能，并在生成长响应时达到154%的速度提升，在准确性和速度方面均优于最先进的VLM加速方法。此外，我们通过引入新型多头twig架构，扩展了TwigVLM 为改进的 TwigVLM++ 变体，该架构包含一个专门的剪枝头。 TwigVLM++ 通过结合蒸馏学习阶段和以剪枝为导向的强化学习阶段的两阶段训练范式，提高了剪枝质量，并进一步通过基于树的SSD策略加速推理。
英文摘要：
Large vision-language models (VLMs) have demonstrated remarkable capabilities in open-world multimodal understanding, yet their high computational overheads pose great challenges for practical deployment. Some recent works have proposed methods to accelerate VLMs by pruning redundant visual tokens guided by the attention maps of VLM's early layers. Despite the success of these token pruning methods, they still suffer from two major shortcomings: (i) considerable accuracy drop due to insensitive attention signals in early layers, and (ii) limited speedup when generating long responses (e.g., 30 tokens). To address the limitations above, we present TwigVLM -- a simple and general architecture by growing a lightweight module, named twig, upon an early layer of the base VLM. Compared with most existing VLM acceleration methods purely based on visual token pruning, our TwigVLM not only achieves better accuracy retention by employing a twig-guided token pruning (TTP) strategy, but also yields higher generation speed by utilizing a self-speculative decoding (SSD) strategy. Taking LLaVA-1.5-7B as the base VLM, experimental results show that TwigVLM preserves 96% of the original performance after pruning 88.9% of the visual tokens and achieves 154% speedup in generating long responses, delivering significantly better performance in terms of both accuracy and speed over the state-of-the-art VLM acceleration methods. Moreover, we extend TwigVLM to an improved TwigVLM++ variant by introducing a novel multi-head twig architecture with a specialized pruning head. TwigVLM++ improves pruning quality via a two-stage training paradigm combining a distillation learning stage and a pruning-oriented reinforcement learning stage, and further accelerates inference via a tree-based SSD strategy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
