利用冻结的单模态编码器实现灵活的多模态对齐
Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
AI总结：
本研究提出利用冻结单模态编码器的多模态对齐框架，通过选语义相似编码器、建概念丰富数据集、训练MLP投影器，以远低于从头训练的成本实现了不错的零样本分类和图文检索效果，提升了多模态开发的可及性与灵活性。
摘要：
近期CLIP等对比式多模态视觉语言模型展现出强大的开放世界语义理解能力，已成为视觉语言应用的标准图像骨干网络。但研究发现训练良好的单模态编码器间存在高语义相似性，由此引出核心问题：是否存在可行方法连接单模态骨干网络以完成视觉语言任务？为此我们提出一种利用冻结单模态编码器实现视觉与语言对齐的新型框架，包括在潜在空间中选择语义相似的编码器、构建富含概念的图像-文本对数据集，以及训练简单的MLP投影器。我们在12个零样本分类数据集和2个图文检索数据集上评估了该方法。我们的最优模型采用DINOv2和All-Roberta-Large文本编码器，在ImageNet上达到76%的准确率，与从头训练的多模态对齐方法相比，数据需求降低20倍，计算需求降低65倍。该框架提升了多模态模型开发的可及性，同时支持在多样场景下灵活适配。代码和构建的数据集可在github.com/mayug/freeze-align获取。
英文摘要：
Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76\(\%\) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at \texttt{github.com/mayug/freeze-align}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
