MMTok: 为VLMs高效推理的多模态覆盖最大化
MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs
AI总结：
MMTok通过多模态覆盖准则提升VLMs推理效率，结合视觉和文本信息优化标记选择，实现性能与速度的平衡。
摘要：
视觉-语言模型（VLMs）通过将视觉输入转换为视觉标记来展示在语言指令下理解视觉内容的出色表现。然而，视觉标记的冗余性导致VLMs的推理效率下降。尽管许多算法已被提出以减少视觉标记的数量，但大多数仅使用单模态信息（即视觉/文本）进行剪枝，并忽略了视觉-语言任务固有的多模态特性。此外，缺乏一个可以应用于不同模态的通用标准。为缓解这一限制，本文提出利用视觉和文本标记通过覆盖标准来选择信息性的视觉标记。我们首先将子集选择问题公式化为最大覆盖问题。随后，一个视觉标记子集被优化以覆盖文本标记和原始视觉标记集合。所提出的方法MMTok在基准数据集上对不同VLMs进行了广泛评估。比较结果表明，视觉和文本信息是互补的，结合多模态信息可以明显超越单模态基线。此外，在POPE数据集上的最大覆盖标准下，我们的方法在保持LLaVA-NeXT-13B原始性能的98.7%的同时实现了1.87倍的速度提升。最后，仅使用四个视觉标记，LLaVA-1.5-7B的原始性能仍能保留87.7%。这些结果突显了覆盖在标记选择中的有效性。代码可在https://github.com/Ironieser/mmtok获取。
英文摘要：
Vision-Language Models (VLMs) demonstrate impressive performance in understanding visual content with language instruction by converting visual inputs to vision tokens. However, redundancy in vision tokens results in the degraded inference efficiency of VLMs. While many algorithms have been proposed to reduce the number of vision tokens, most of them apply only unimodal information (i.e., vision/text) for pruning and ignore the inherent multimodal property of vision-language tasks. Moreover, it lacks a generic criterion that can be applied to different modalities. To mitigate this limitation, in this work, we propose to leverage both vision and text tokens to select informative vision tokens by the coverage criterion. We first formulate the subset selection problem as a maximum coverage problem. Afterwards, a subset of vision tokens is optimized to cover the text tokens and the original set of vision tokens, simultaneously. The proposed method MMTok is extensively evaluated on benchmark datasets with different VLMs. The comparison illustrates that vision and text information are complementary, and combining multimodal information can surpass the unimodal baseline with a clear margin. Moreover, under the maximum coverage criterion on the POPE dataset, our method achieves a 1.87x speedup while maintaining 98.7% of the original performance on LLaVA-NeXT-13B. Finally, with only four vision tokens, 87.7% of the original performance is still preserved on LLaVA-1.5-7B. These results highlight the effectiveness of coverage in token selection. The code is available at https://github.com/Ironieser/mmtok
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
