用于高效视觉语言模型的无注意力轻量级令牌约简
Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
浏览论文内容
中文总结 AI 辅助
针对视觉语言模型在边缘设备部署难的问题，提出无注意力轻量级令牌约简框架，通过基于熵的准则估计令牌重要性，引入一致性信号确保视觉覆盖，经实验验证该框架在精度和效率权衡上表现出色。
中文摘要 AI 辅助
视觉语言模型（VLMs）在多模态理解方面表现出色，但由于处理大量视觉令牌的计算开销大，在资源受限的边缘设备上部署仍具挑战。令牌约简是加速VLMs推理的一个有前景的方向，但现有方法存在问题。本文提出一个无注意力轻量级令牌约简框架作为VLMs的即插即用模块，保留重要且多样的令牌以生成紧凑视觉表示。首先，采用信息论视角，用基于熵的新准则量化令牌信息进行无注意力重要性估计。其次，引入变换诱导一致性信号以轻量级方式确保多样视觉覆盖。实验表明该框架在精度-效率权衡方面表现良好。
英文摘要
Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.
发表机构
- Zhejiang University（浙江大学）
- Hong Kong University of Science and Technology（香港科技大学）
机构由 AI 辅助整理，请以论文原文为准。
