arXiv 2405.190092024-06-14cs.CV

通过未掩码 Token 对齐增强视觉-语言模型

Enhancing Vision-Language Model with Unmasked Token Alignment

Institute for AI Industry Research (AIR)（人工智能产业研究院）；Tsinghua University（清华大学）；Sensetime Research（商汤科技研究院）；CPII under InnoHK（InnoHK下属CPII）；Shanghai AI Laboratory（上海人工智能实验室）

Jihao Liu, Jinliang Zheng, Boxiao Liu, Yu Liu, Hongsheng Li

备注：: Accepted by TMLR; Code and models are available at https://github.com/jihaonew/UTA

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 UTA，通过将未掩码视觉 token 对齐到冻结 CLIP 视觉编码器来训练 ViT，高效增强视觉-语言表示，并在多个单模态与多模态基准上超过 MIM 方法。

摘要：

以 CLIP 为代表的图像-文本对对比预训练，已成为学习多模态视觉-语言表示的标准技术。尽管 CLIP 展现出显著性能，但在噪声网络规模数据集上从头训练它在计算上代价高昂。另一方面，先掩码再预测的预训练方法（如 Masked Image Modeling，MIM）为单模态表示提供了高效的自监督学习。本文提出 Unmasked Token Alignment（UTA，未掩码 Token 对齐），该方法利用现有 CLIP 模型进一步增强其视觉-语言表示。UTA 通过将未掩码视觉 token 与来自冻结 CLIP 视觉编码器的相应图像 token 对齐来训练 Vision Transformer（ViT），从而使 ViT 模型自动与 CLIP 文本编码器对齐。预训练后的 ViT 即使没有在图像-文本对上训练，也可直接用于零样本评估。与 MIM 方法相比，UTA 不存在训练与微调不一致的问题，并且通过避免使用额外的 [MASK] token，训练效率大幅提高。大量实验结果表明，UTA 能够增强 CLIP 模型，并在多种单模态和多模态基准上优于现有 MIM 方法。代码和模型可在 https://github.com/jihaonew/UTA 获取。

英文摘要：

Contrastive pre-training on image-text pairs, exemplified by CLIP, becomes a standard technique for learning multi-modal visual-language representations. Although CLIP has demonstrated remarkable performance, training it from scratch on noisy web-scale datasets is computationally demanding. On the other hand, mask-then-predict pre-training approaches, like Masked Image Modeling (MIM), offer efficient self-supervised learning for single-modal representations. This paper introduces Unmasked Token Alignment (UTA), a method that leverages existing CLIP models to further enhance its vision-language representations. UTA trains a Vision Transformer (ViT) by aligning unmasked visual tokens to the corresponding image tokens from a frozen CLIP vision encoder, which automatically aligns the ViT model with the CLIP text encoder. The pre-trained ViT can be directly applied for zero-shot evaluation even without training on image-text pairs. Compared to MIM approaches, UTA does not suffer from training-finetuning inconsistency and is much more training-efficient by avoiding using the extra [MASK] tokens. Extensive experimental results demonstrate that UTA can enhance CLIP models and outperform existing MIM methods on various uni- and multi-modal benchmarks. Code and models are available at https://github.com/jihaonew/UTA.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。