CoME-VL：扩展互补多编码视觉语言学习
CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learning
AI总结：
本文提出CoME-VL框架，通过融合对比学习和自监督编码器提升视觉语言模型性能，实验显示在多个基准上优于单编码基线。
摘要：
近期视觉语言模型（VLMs）通常依赖单一视觉编码器，通过对比图像-文本目标进行预训练，如CLIP式预训练。尽管对比编码器在跨模态对齐和检索中有效，自监督视觉编码器通常能捕获更丰富的密集语义并在识别和理解任务中表现更稳健。本文研究如何扩展这些互补视觉表示的融合。我们提出CoME-VL：互补多编码视觉语言模型，一种模块化融合框架，整合了对比训练的视觉编码器和自监督DINO编码器。我们的方法通过（i）熵引导的多层聚合与正交约束投影减少冗余，以及（ii）RoPE增强的交叉注意力对齐异构标记网格并生成紧凑融合视觉标记。这些融合标记可以注入到解码器-only LLM中，对标准VLM流程进行最小修改。在多样化的视觉语言基准上进行了广泛的实验，证明CoME-VL在多个基准上一致优于单编码基线。特别是，在视觉理解任务上平均提升4.9%，在接地任务上提升5.4%。我们的方法在RefCOCO检测任务上达到最佳性能，同时大幅超越基线。最后，我们进行了层合并、非冗余特征混合和融合能力的消融研究，评估互补对比学习和自监督信号如何影响VLM性能。
英文摘要：
Recent vision-language models (VLMs) typically rely on a single vision encoder trained with contrastive image-text objectives, such as CLIP-style pretraining. While contrastive encoders are effective for cross-modal alignment and retrieval, self-supervised visual encoders often capture richer dense semantics and exhibit stronger robustness on recognition and understanding tasks. In this work, we investigate how to scale the fusion of these complementary visual representations for vision-language modeling. We propose CoME-VL: Complementary Multi-Encoder Vision-Language, a modular fusion framework that integrates a contrastively trained vision encoder with a self-supervised DINO encoder. Our approach performs representation-level fusion by (i) entropy-guided multi-layer aggregation with orthogonality-constrained projections to reduce redundancy, and (ii) RoPE-enhanced cross-attention to align heterogeneous token grids and produce compact fused visual tokens. The fused tokens can be injected into a decoder-only LLM with minimal changes to standard VLM pipelines. Extensive experiments across diverse vision-language benchmarks demonstrate that CoME-VL consistently outperforms single-encoder baselines. In particular, we observe an average improvement of 4.9% on visual understanding tasks and 5.4% on grounding tasks. Our method achieves state-of-the-art performance on RefCOCO for detection while improving over the baseline by a large margin. Finally, we conduct ablation studies on layer merging, non-redundant feature mixing, and fusion capacity to evaluate how complementary contrastive and self-supervised signals affect VLM performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
