QMoP：基于查询的混合投影器用于高效的视觉令牌压缩
QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compression
AI总结：
本文提出QMoP框架，通过三种协作分支实现视觉令牌的自适应压缩，结合查询引导路由和专家融合机制，提升效率并减少资源消耗。
摘要：
多模态大语言模型面临严重的计算和内存瓶颈，因为视觉令牌数量远超文本令牌。尽管最近的方法使用投影模块对齐和压缩视觉令牌，但通常依赖固定启发式方法，限制了在多样化场景中的适应性。本文首先提出查询引导的混合投影器（QMoP），一种新颖且灵活的框架，通过三个协作分支自适应压缩视觉令牌：（1）基于池化的分支用于粗粒度全局语义，（2）重新采样分支用于提取高层语义表示，（3）基于剪枝的分支用于精细的令牌选择以保留关键视觉细节。为了自适应协调这些分支，我们引入查询引导路由（QGR），根据视觉输入和文本查询动态选择和加权不同分支的输出。设计了一种专家混合风格的融合机制，以聚合输出，利用每种策略的优势并抑制噪声。为了系统评估视觉令牌压缩的影响，我们还开发了VTCBench，一个专门用于评估压缩引起的信源损失的基准。大量实验表明，尽管依赖于基本压缩模块，QMoP在强基线之上表现更优，并在内存、计算和推理时间上实现了显著节省。
英文摘要：
Multimodal large language models suffer from severe computational and memory bottlenecks, as the number of visual tokens far exceeds that of textual tokens. While recent methods employ projector modules to align and compress visual tokens into text-aligned features, they typically depend on fixed heuristics that limit adaptability across diverse scenarios. In this paper, we first propose Query Guided Mixture-of-Projector (QMoP), a novel and flexible framework that adaptively compresses visual tokens via three collaborative branches: (1) a pooling-based branch for coarse-grained global semantics, (2) a resampler branch for extracting high-level semantic representations, and (3) a pruning-based branch for fine-grained token selection to preserve critical visual detail. To adaptively coordinate these branches, we introduce the Query Guided Router (QGR), which dynamically selects and weights the outputs from different branches based on both visual input and textual queries. A Mixture-of-Experts-style fusion mechanism is designed to aggregate the outputs, harnessing the strengths of each strategy while suppressing noise. To systematically evaluate the effects of Visual Token Compression, we also develop VTCBench, a dedicated benchmark for evaluating the information loss induced by visual token compression. Extensive experiments demonstrate that despite relying on fundamental compression modules, QMoP outperforms strong baselines and delivers significant savings in memory, computation, and inference time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
