通过格罗莫夫-瓦瑟斯坦距离重新思考VLM中的模型选择
Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
AI总结：
本文通过系统实验探讨视觉编码器选择的关键因素，发现模态间结构相似性通过格罗莫夫-瓦瑟斯坦距离衡量能提升VLM性能预测。
摘要：
视觉语言模型（VLMs）通过整合视觉编码器增强了传统大语言模型（LLMs）的视觉能力。尽管近期研究探索了多种视觉编码器与LLMs的组合，但缺乏对何种视觉编码器适合VLM对齐的系统理解。本文通过在精心整理的19个预训练视觉编码器集合上进行全面实验，系统研究这一问题。我们发现，常见的选择策略，如选择编码器最大尺寸或最高零样本准确度，始终无法识别最优模型。事实上，这些指标与VLM性能的相关性较弱至中等。这一发现引发了根本性问题：哪些视觉编码器因素对VLM至关重要？通过全面分析，我们发现模态间结构相似性在视觉编码器选择中起关键作用，我们使用格罗莫夫-瓦瑟斯坦距离作为代理进行测量。从理论角度看，我们证明了跨模态映射的可学习性可以被证明与格罗莫夫-瓦瑟斯坦距离相关联。在60多个完整的VLM训练运行中，我们的所提推理-only度量显著优于替代模型选择策略，并表现出与最终VLM性能更强的相关性，从而在完整训练前实现了高效的VLM性能预测。
英文摘要：
Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders. While recent works have explored various combinations of vision encoders and LLMs, there still lacks a principled understanding of what makes a vision encoder suitable for VLM alignment. In this paper, we systematically investigate this question via comprehensive experiments on a curated collection of 19 pre-trained vision encoders from diverse sources. We first demonstrate that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models. In fact, these metrics show only weak to moderate correlation with VLM performance. This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM? Through comprehensive analysis, we identify that the structural similarity across modalities plays a crucial but previously overlooked role in vision-encoder selection, which we measure using the Gromov-Wasserstein distance as a proxy. From a theoretical perspective, we show that the learnability of cross-modality mapping can be provably associated with the Gromov-Wasserstein distance. Empirical verification on 60+ full VLM training runs shows that our proposed inference-only metric performs significantly better than alternative model selection strategies and exhibits a much stronger correlation with final VLM performance, thereby enabling efficient and effective prediction of VLM performance before full training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
