CoAVT:一种受认知启发的统一音频-视觉-文本预训练模型,用于多模态处理
CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing
AI总结:
本文提出受认知启发的CoAVT模型,通过联合音频-视觉编码器、文本编码器和查询编码器,结合三模态对齐与双模态对齐,在多模态预训练中取得AudioCaps、AudioSet和VGGSound等任务上的最先进性能。
摘要:
长期以来,人们一直追求一种统一的音频-视觉-文本模型,以实现各种多模态理解任务,这模仿了人类的听、看和读的过程。人类倾向于使用两个独立的系统来表示知识:一个用于表示言语(文本)信息,另一个用于表示非言语(视觉和听觉)信息。这两个系统可以独立运作,但也可以相互作用。受这种对人类认知理解的启发,在本文中,我们提出了CoAVT——一种新颖的受认知启发的相关音频-视觉-文本预训练模型,用于连接这三种模态。它包含一个联合音频-视觉编码器,该编码器学习将音频-视觉同步信息与音频和视觉内容一起编码以表示非言语信息,以及一个文本编码器用于处理言语信息的文本输入。为了弥合模态之间的差距,CoAVT采用了一个查询编码器,其中包含一组可学习的查询嵌入,并提取相应文本中最具信息量的视听特征。此外,为了分别利用音频与语言以及视觉与语言之间的对应关系,我们还在基础的视听-文本三模态对齐之上建立了音频-文本和视觉-文本双模态对齐,以增强多模态表示学习。最后,我们使用三个多模态目标联合优化CoAVT模型:对比损失、匹配损失和语言建模损失。大量实验表明,CoAVT能够学习强大的多模态相关性,并推广到各种下游任务。CoAVT在AudioCaps上的文本-视频检索任务中,在零样本和微调设置下均取得了新的最先进性能,并在AudioSet和VGGSound上的音频-视觉事件分类和音频-视觉检索任务中也取得了新的最先进性能。
英文摘要:
There has been a long-standing quest for a unified audio-visual-text model to enable various multimodal understanding tasks, which mimics the listening, seeing and reading process of human beings. Humans tends to represent knowledge using two separate systems: one for representing verbal (textual) information and one for representing non-verbal (visual and auditory) information. These two systems can operate independently but can also interact with each other. Motivated by this understanding of human cognition, in this paper, we introduce CoAVT -- a novel cognition-inspired Correlated Audio-Visual-Text pre-training model to connect the three modalities. It contains a joint audio-visual encoder that learns to encode audio-visual synchronization information together with the audio and visual content for non-verbal information, and a text encoder to handle textual input for verbal information. To bridge the gap between modalities, CoAVT employs a query encoder, which contains a set of learnable query embeddings, and extracts the most informative audiovisual features of the corresponding text. Additionally, to leverage the correspondences between audio and vision with language respectively, we also establish the audio-text and visual-text bi-modal alignments upon the foundational audiovisual-text tri-modal alignment to enhance the multimodal representation learning. Finally, we jointly optimize CoAVT model with three multimodal objectives: contrastive loss, matching loss and language modeling loss. Extensive experiments show that CoAVT can learn strong multimodal correlations and be generalized to various downstream tasks. CoAVT establishes new state-of-the-art performance on text-video retrieval task on AudioCaps for both zero-shot and fine-tuning settings, audio-visual event classification and audio-visual retrieval tasks on AudioSet and VGGSound.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。