视觉与语言编码器对世界的表示是否相似？
Do Vision and Language Encoders Represent the World Similarly?
AI总结：
本文研究了单模态视觉与语言编码器表示同一物理世界时的潜在对齐性，提出基于CKA和种子图匹配的无训练方法，在跨语言、跨域字幕匹配等任务上验证了其有效性。
摘要：
诸如CLIP等对齐的文本-图像编码器已成为视觉-语言任务的事实标准模型。此外，特定模态的编码器在各自领域也取得了令人瞩目的性能。这引发了一个核心问题：既然单模态视觉和语言编码器从根本上表示的是同一个物理世界，它们之间是否存在对齐？我们使用中心化核对齐（CKA）分析图像-字幕基准上视觉和语言模型的潜在空间结构，发现未对齐与已对齐编码器的表示空间在语义上是相似的。在CLIP等已对齐编码器中缺乏统计相似性的情况下，我们表明未对齐编码器之间无需任何训练即可存在可能的匹配。我们将其构建为一个利用图之间语义相似性的种子图匹配问题，并提出了两种方法——快速二次分配问题优化，以及一种基于局部CKA度量的匹配/检索。我们在多个下游任务上证明了该方法的有效性，包括跨语言、跨域字幕匹配和图像分类。代码可在github.com/mayug/0-shot-llm-vision获取。
英文摘要：
Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. Furthermore, modality-specific encoders achieve impressive performances in their respective domains. This raises a central question: does an alignment exist between uni-modal vision and language encoders since they fundamentally represent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training. We frame this as a seeded graph-matching problem exploiting the semantic similarity between graphs and propose two methods - a Fast Quadratic Assignment Problem optimization, and a novel localized CKA metric-based matching/retrieval. We demonstrate the effectiveness of this on several downstream tasks including cross-lingual, cross-domain caption matching and image classification. Code available at github.com/mayug/0-shot-llm-vision.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
