arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07543cs.CV
收藏

基于潜在压缩学习的交错图文数据视觉模型预训练

Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning

Tsinghua University(清华大学);OpenGVLab(开放视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);The Chinese University of Hong Kong(香港中文大学);University of Science and Technology of China(中国科学技术大学);Xi’an Jiaotong University(西安交通大学);Beijing University of Posts and Telecommunications(北京邮电大学)

Chenyu Yang, Xizhou Zhu, Jinguo Zhu, Weijie Su, Junjie Wang, Xuan Dong, Wenhai Wang, Lewei Lu, Bin Li, Jie Zhou, Yu Qiao, Jifeng Dai

AI总结:

针对互联网普遍存在的交错图文数据未被有效利用的问题,提出潜在压缩学习(LCL)方法,通过最大化因果注意力模型输入输出的互信息进行预训练,性能匹配CLIP并从头学到鲁棒视觉表征。

摘要:

最近,视觉模型预训练已经从依赖人工标注的数据集发展到利用大规模网络抓取的图文数据。尽管取得了这些进展,但目前还没有一种预训练方法能够有效利用在互联网上非常普遍的交错图文数据。受自然语言处理中压缩学习近期取得成功的启发,我们提出了一种针对交错图文数据的全新视觉模型预训练方法,称为 Latent Compression Learning (LCL)。该方法通过最大化因果注意力模型输入与输出之间的互信息来执行潜在压缩学习。训练目标可分解为两个基本任务:1)视觉表征与上文上下文之间的对比学习;2)基于视觉表征生成下文文本。我们的实验表明,我们的方法不仅在配对预训练数据集(如 LAION)上匹配了 CLIP 的性能,还能利用交错预训练数据(如 MMC4)从头学习鲁棒的视觉表征,展示了利用交错图文数据进行视觉模型预训练的潜力。代码已发布于 https://github.com/OpenGVLab/LCL。

英文摘要:

Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representation from scratch, showcasing the potential of vision model pre-training with interleaved image-text data. Code is released at https://github.com/OpenGVLab/LCL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑