轻踩油门：重新审视用于视觉-语言模型加速的视觉令牌剪枝
Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
AI总结：
本研究发现现有视觉令牌早期剪枝加速法存在图像顶部令牌过度剪枝的问题，且现有基准难以检测该缺陷，据此提出FEATHER多阶段剪枝方法，在同等计算节省下将定位基准性能提升5倍以上。
摘要：
近期关于视觉-语言模型加速的研究尽管高度压缩了视觉信息，仍在多种视觉-语言任务上取得了优异性能。本研究调研了在语言模型内部对视觉令牌进行早期剪枝的主流加速方法。令人意外的是，我们发现该方法虽在多数任务上保持了良好性能，但在定位等部分以视觉为核心的任务子集上表现出截然不同的行为。进一步研究发现，该加速方法存在一个核心问题：图像顶部的大部分令牌都会被剪枝掉。然而，在许多旨在评估以视觉为核心的能力的基准测试中，这种存在缺陷的剪枝策略仍能保持不错的性能，凸显出这些基准测试评估细粒度视觉能力的能力有限。基于这些发现，我们提出FEATHER（基于集成准则的快速有效加速），这一简洁方法解决了所发现的早期层剪枝问题，并通过结合早期均匀采样的多阶段剪枝进一步增强相关令牌的保留度，以确保广泛的图像覆盖。在计算节省量相当的情况下，我们发现FEATHER在以视觉为核心的定位基准测试上的性能相比原始加速方法提升了5倍以上。
英文摘要：
Recent works on accelerating Vision-Language Models achieve strong performance across a variety of vision-language tasks despite highly compressing visual information. In this work, we examine the popular acceleration approach of early pruning of visual tokens inside the language model. Surprisingly, we find that while strong performance is maintained across many tasks, it exhibits drastically different behavior for a subset of vision-centric tasks such as localization. Upon further investigation, we uncover a core issue with the acceleration approach where most tokens towards the top of the image are pruned away. Yet, on many benchmarks aiming to evaluate vision-centric capabilities, strong performance persists with the flawed pruning strategy, highlighting these benchmarks' limited ability to assess fine-grained visual capabilities. Based on these findings, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble cRiteria), a straightforward approach that resolves the discovered early-layer pruning issue and further enhances the preservation of relevant tokens via multistage pruning with early uniform sampling to ensure broad image coverage. With comparable computational savings, we find that FEATHER achieves more than 5x performance improvement on the vision-centric localization benchmarks compared to the original acceleration approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
