arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01852cs.CVcs.AI
收藏

LanguageBind:通过基于语言的语义对齐将视频-语言预训练扩展到 N 模态

LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment

Peking University(北京大学);Tencent Data Platform(腾讯数据平台);Pengcheng Lab(鹏城实验室);National University of Singapore(新加坡国立大学);Nari Technology Development Limited Company(南瑞集团有限公司)

Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan

AI总结:

针对视频-语言预训练难以扩展到 N 模态的问题,本文提出 LanguageBind,以语言为纽带通过冻结语言编码器和对比学习对齐多模态,并构建 VIDAL-10M 数据集,在 15 个基准上验证有效性。

摘要:

视频-语言(VL)预训练已在多个下游任务上取得显著提升。然而,当前的 VL 预训练框架很难扩展到视觉和语言之外的多种模态(N 种模态,N>=3)。因此,我们提出 LanguageBind,将语言作为连接不同模态的纽带,因为语言模态已得到充分研究并包含丰富语义。具体而言,我们冻结通过 VL 预训练获得的语言编码器,然后利用对比学习训练其他模态的编码器。由此,所有模态都被映射到一个共享特征空间,实现多模态语义对齐。尽管 LanguageBind 确保我们能够将 VL 模态扩展到 N 种模态,但我们还需要一个以语言为中心、包含对齐数据对的高质量数据集。因此,我们提出包含视频、红外、深度、音频及其对应语言的 VIDAL-10M,并将其命名为 VIDAL-10M。在我们的 VIDAL-10M 中,所有视频均来自具有完整语义的短视频平台,而不是长视频中被截断的片段,并且所有视频、深度、红外和音频模态都与其文本描述对齐。LanguageBind 在涵盖视频、音频、深度和红外的广泛 15 个基准测试上取得了优异性能。此外,多项实验证明了 LanguageBind 在实现不同模态之间间接对齐和互补性方面的有效性。代码地址:https://github.com/PKU-YuanGroup/LanguageBind

英文摘要:

The video-language (VL) pretraining has achieved remarkable improvement in multiple downstream tasks. However, the current VL pretraining framework is hard to extend to multiple modalities (N modalities, N>=3) beyond vision and language. We thus propose LanguageBind, taking the language as the bind across different modalities because the language modality is well-explored and contains rich semantics. Specifically, we freeze the language encoder acquired by VL pretraining, then train encoders for other modalities with contrastive learning. As a result, all modalities are mapped to a shared feature space, implementing multi-modal semantic alignment. While LanguageBind ensures that we can extend VL modalities to N modalities, we also need a high-quality dataset with alignment data pairs centered on language. We thus propose VIDAL-10M with Video, Infrared, Depth, Audio and their corresponding Language, naming as VIDAL-10M. In our VIDAL-10M, all videos are from short video platforms with complete semantics rather than truncated segments from long videos, and all the video, depth, infrared, and audio modalities are aligned to their textual descriptions. LanguageBind has achieved superior performance on a wide range of 15 benchmarks covering video, audio, depth, and infrared. Moreover, multiple experiments have provided evidence for the effectiveness of LanguageBind in achieving indirect alignment and complementarity among diverse modalities. Code address: https://github.com/PKU-YuanGroup/LanguageBind

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑