基于多层特征聚合的MLLMs空间感知高效投影器
Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation
AI总结：
针对多模态语言模型中视觉与语言token序列的空间差异问题，提出空间感知高效投影器（SAEP），利用改进的深度卷积增强多层视觉特征空间信息，在将视觉token大幅减少75%的同时显著提升了多模态理解性能。
摘要：
投影器在多模态语言模型（MLLMs）中扮演着至关重要的角色。其输出的视觉token数量影响MLLM的效率，而视觉token的质量则影响MLLM的视觉理解能力。当前对投影器的探索主要集中在减少视觉token数量以提高效率，往往忽略了序列化的2维视觉token序列与自然语言token序列之间固有的空间差异。为解决这一问题，提出了一种空间感知高效投影器（SAEP）。具体而言，我们的SAEP方法在多层视觉特征上采用了一个改进的可分离深度卷积模块，以增强视觉token的空间信息。结果表明，我们的SAEP方法不仅能将视觉token数量大幅减少75%，还能显著提升MLLM的多模态空间理解能力。此外，与现有投影器相比，我们的SAEP在大量多模态评估基准上取得了最佳性能，这表明其在弥合模态鸿沟方面的有效性。
英文摘要：
The projector plays a crucial role in multi-modal language models (MLLMs). The number of visual tokens it outputs affects the efficiency of the MLLM, while the quality of the visual tokens influences the visual understanding capabilities of the MLLM. Current explorations on the projector focus on reducing the number of visual tokens to improve efficiency, often overlooking the inherent spatial discrepancy between the serialized 2-dimensional visual token sequences and natural language token sequences. A Spatial-Aware Efficient Projector (SAEP) is proposed to address this issue. In detail, our SAEP method employs an modified separable depthwise convolution module on multi-layer visual features to enhance the spatial information of visual tokens. As a result, our SAEP method can not only largely reduce the number of visual tokens by 75\%, but also significantly improve the multimodal spatial understanding capability of MLLMs. Moreover, compared to existing projectors, our SAEP gets best performances on massive multimodal evaluation benchmarks, which denotes its effectiveness on bridging the modality gap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
