arXiv 2304.083452025-01-06cs.LGcs.CLcs.CVcs.MMeess.AS

VALOR：视觉-音频-语言全感知预训练模型与数据集

VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset

School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University of Science and Technology（南京理工大学）

Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, Jinhui Tang

备注：: Preprint version w/o audio files embeded in PDF. Audio embeded version can be found on project page or github

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出端到端视觉-音频-语言预训练模型 VALOR，通过 MGA、MGC 任务和 VALOR-1M 数据集学习三模态对齐与条件文本生成，并在多项跨模态基准上达到最优。

摘要：

本文提出一种用于多模态理解与生成的视觉-音频-语言全感知预训练模型 VALOR。与被广泛研究的视觉-语言预训练模型不同，VALOR 以端到端方式联合建模视觉、音频和语言之间的关系。它包含三个相互独立的编码器，用于获得单模态表示，以及一个用于多模态条件文本生成的解码器。我们设计了两个预训练任务来预训练 VALOR 模型，即多模态分组对齐（MGA）和多模态分组描述（MGC）。MGA 将视觉、语言和音频投影到同一公共空间，同时建立视觉-语言、音频-语言以及视听-语言对齐。MGC 学习如何在视觉、音频或二者共同作为条件时生成文本 token。为推动视觉-音频-语言预训练研究，我们构建了名为 VALOR-1M 的大规模高质量三模态数据集，其中包含 100 万个带有人工标注视听描述的可发声视频。大量实验表明，VALOR 能够学习到强多模态关联，并可泛化到检索、描述和问答等多种下游任务，适应视觉-语言、音频-语言和视听-语言等不同输入模态。VALOR 在一系列公共跨模态基准上取得了新的最优性能。代码和数据可在项目页面 https://casia-iva-group.github.io/projects/VALOR 获取。

英文摘要：

In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https://casia-iva-group.github.io/projects/VALOR.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。