单模态视觉与语言模型的对齐评估与学习
Assessing and Learning Alignment of Unimodal Vision and Language Models
AI总结：
本文提出直接评估方法并引入SAIL框架，利用预训练单模态模型高效对齐视觉与语言，仅需6%配对数据，在ImageNet零样本准确率达73.4%，优于CLIP，并提升多模态大语言模型性能。
摘要：
单模态视觉和语言模型的对齐程度如何？尽管先前的工作已经尝试回答这个问题，但他们的评估方法并不能直接转化到这些模型在实际视觉-语言任务中的使用方式上。在本文中，我们提出了一种直接评估方法，受线性探测启发，用于评估视觉-语言对齐。我们识别出SSL视觉模型的对齐程度取决于其SSL训练目标，并发现SSL表示的聚类质量比其线性可分性对齐性能的影响更强。接下来，我们引入了图像和语言的快速对齐（SAIL），一个高效的迁移学习框架，用于对齐预训练的单模态视觉和语言模型以应对下游视觉-语言任务。由于SAIL利用了预训练单模态模型的优势，它只需要显著更少的（6%）配对图像-文本数据用于多模态对齐，相比像CLIP这样从头训练的模型。SAIL训练仅需要单个A100 GPU，5小时的训练时间，并且可以容纳高达32,768的批大小。SAIL在ImageNet上实现了73.4%的零样本准确率（而CLIP为72.7%），并在零样本检索、复杂推理和语义分割方面表现出色。此外，SAIL提高了视觉编码器的语言兼容性，进而增强了多模态大语言模型的性能。整个代码库和模型权重是开源的：https://lezhang7.github.io/sail.github.io/
英文摘要：
How well are unimodal vision and language models aligned? Although prior work have approached answering this question, their assessment methods do not directly translate to how these models are used in practical vision-language tasks. In this paper, we propose a direct assessment method, inspired by linear probing, to assess vision-language alignment. We identify that the degree of alignment of the SSL vision models depends on their SSL training objective, and we find that the clustering quality of SSL representations has a stronger impact on alignment performance than their linear separability. Next, we introduce Swift Alignment of Image and Language (SAIL), a efficient transfer learning framework that aligns pretrained unimodal vision and language models for downstream vision-language tasks. Since SAIL leverages the strengths of pretrained unimodal models, it requires significantly fewer (6%) paired image-text data for the multimodal alignment compared to models like CLIP which are trained from scratch. SAIL training only requires a single A100 GPU, 5 hours of training and can accommodate a batch size up to 32,768. SAIL achieves 73.4% zero-shot accuracy on ImageNet (vs. CLIP's 72.7%) and excels in zero-shot retrieval, complex reasoning, and semantic segmentation. Additionally, SAIL improves the language-compatibility of vision encoders that in turn enhance the performance of multimodal large language models. The entire codebase and model weights are open-source: https://lezhang7.github.io/sail.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
