AdaFV：重新思考视觉-语言对齐以加速 VLM
AdaFV: Rethinking of Visual-Language alignment for VLM acceleration
AI总结：
本文提出 AdaFV，一种自适应跨模态注意力混合机制，通过在 pre-LLM 层动态结合视觉显著性和文本-图像相似性选择信息量大的视觉 token，实现无需训练的 VLM 加速，在高缩减率下达到 SOTA 性能。
摘要：
VLMs 的成功通常依赖于动态高分辨率模式，该模式自适应地将输入图像增强为多个裁剪块，从而保留图像的细节。然而，这种方法会产生大量冗余的视觉 token，从而显著降低 VLMs 的效率。为了在不引入额外训练成本的情况下提高 VLMs 的效率，许多研究工作通过过滤无信息量的视觉 token 或聚合其信息来减少视觉 token。一些方法提出根据 VLMs 的自注意力来减少视觉 token，但这存在偏差，会导致响应不准确。仅依赖视觉线索的 token 减少方法是与文本无关的，无法聚焦于与问题最相关的区域，特别是当查询对象在图像中不显著时。在本工作中，我们首先进行实验，表明原始文本嵌入与视觉 token 是对齐的，对尾部视觉 token 没有偏差。然后，我们提出了一种自适应跨模态注意力混合机制，在 pre-LLM 层中动态利用视觉显著性和文本到图像相似性的有效性，以选择具有信息量的视觉 token。大量实验表明，所提出的方法实现了最先进的无训练 VLM 加速性能，特别是在缩减率足够大时。
英文摘要：
The success of VLMs often relies on the dynamic high-resolution schema that adaptively augments the input images to multiple crops, so that the details of the images can be retained. However, such approaches result in a large number of redundant visual tokens, thus significantly reducing the efficiency of the VLMs. To improve the VLMs' efficiency without introducing extra training costs, many research works are proposed to reduce the visual tokens by filtering the uninformative visual tokens or aggregating their information. Some approaches propose to reduce the visual tokens according to the self-attention of VLMs, which are biased, to result in inaccurate responses. The token reduction approaches solely rely on visual cues are text-agnostic, and fail to focus on the areas that are most relevant to the question, especially when the queried objects are non-salient to the image. In this work, we first conduct experiments to show that the original text embeddings are aligned with the visual tokens, without bias on the tailed visual tokens. We then propose a self-adaptive cross-modality attention mixture mechanism that dynamically leverages the effectiveness of visual saliency and text-to-image similarity in the pre-LLM layers to select the visual tokens that are informative. Extensive experiments demonstrate that the proposed approach achieves state-of-the-art training-free VLM acceleration performance, especially when the reduction rate is sufficiently large.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
