arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.09582cs.CV
收藏

NEVLP:用于高效视觉-语言预训练的噪声鲁棒框架

NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training

Johns Hopkins University(约翰斯·霍普金斯大学);University of California, Berkeley(加州大学伯克利分校);University of California San Diego(加州大学圣地亚哥分校);Duke University(杜克大学)

Yiyi Tao, Zhuoyue Wang, Hang Zhang, Lun Wang

AI总结:

本文提出NEVLP框架,通过transformer桥接冻结图像编码器与大型语言模型,并采用噪声自适应正则化和概念增强学习策略,有效缓解网络数据噪声与不完整问题,以更少预训练数据实现视觉语言任务的最先进性能。

摘要:

视觉语言模型(VLMs)在各种视觉-语言任务上的成功,严重依赖于使用大规模网络抓取数据集进行预训练。然而,网络数据具有噪声和不完整的特性,使得数据集规模对性能至关重要,导致端到端训练变得日益不可行。在本文中,我们提出了NEVLP,这是一个用于高效视觉-语言预训练的噪声鲁棒框架,它需要更少的预训练数据。具体而言,我们通过一个transformer桥接了冻结的图像编码器与大型语言模型之间的模态鸿沟,并引入了两种创新的学习策略:噪声自适应学习和概念增强学习,以减轻噪声的影响。在噪声自适应学习中,我们基于transformer的记忆效应估计每个图文对的噪声概率,并在图文对比学习上采用噪声自适应正则化来约束跨模态对齐。在概念增强学习中,我们通过引入视觉概念(图像中的对象)来丰富不完整的文本,为图文匹配和基于图像的文本生成提供关于现有对象的先验信息,从而缓解文本不完整的问题。我们的框架有效利用了含噪网络数据,并在广泛的视觉-语言任务(包括图文检索、图像描述和视觉问答)中,以更少的预训练数据实现了最先进的性能。

英文摘要:

The success of Vision Language Models (VLMs) on various vision-language tasks heavily relies on pre-training with large scale web-crawled datasets. However, the noisy and incomplete nature of web data makes dataset scale crucial for performance, rendering end-to-end training increasingly prohibitive. In this paper, we propose NEVLP, a noise-robust framework for efficient vision-language pre-training that requires less pre-training data. Specifically, we bridge the modality gap between a frozen image encoder and a large language model with a transformer and introduce two innovative learning strategies: noise-adaptive learning and concept-enhanced learning to mitigate the impact of noise. In noise-adaptive learning, we estimate the noise probability of each image-text pair based on the transformer's memorization effect and employ noise-adaptive regularization on image-text contrastive learning to condition cross-modal alignment. In concept-enhanced learning, we enrich incomplete text by incorporating visual concepts (objects in the image) to provide prior information about existing objects for image-text matching and image-grounded text generation, thereby mitigating text incompletion. Our framework effectively utilizes noisy web data and achieves state-of-the-art performance with less pre-training data across a wide range of vision-language tasks, including image-text retrieval, image captioning, and visual question answering.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑