arXiv 2210.078392023-04-11cs.MMcs.CVcs.SDeess.AS

对比式音视频掩码自编码器

Contrastive Audio-Visual Masked Autoencoder

MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；UT Austin（德克萨斯大学奥斯汀分校）；IBM Research AI（IBM研究院人工智能部门）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Goethe University Frankfurt（法兰克福大学）

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass

备注：: Accepted at ICLR 2023 as a notable top 25% paper. Code and pretrained models are at https://github.com/yuangongnd/cav-mae

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CAV-MAE，将MAE扩展到音视频多模态，并融合对比学习与掩码数据建模，实现完全自监督的联合音视频表示学习，在VGGSound上达到65.9%的新SOTA准确率。

摘要：

在本文中，我们首先将最近的掩码自编码器（MAE）模型从单一模态扩展到音视频多模态。随后，我们通过结合对比学习和掩码数据建模这两种主要的自监督学习框架，提出了对比式音视频掩码自编码器（CAV-MAE），以学习联合且协调的音视频表示。我们的实验表明，对比式音视频对应学习目标不仅使模型能够执行音视频检索任务，而且有助于模型学习更好的联合表示。因此，我们完全自监督预训练的CAV-MAE在VGGSound上取得了65.9%的新SOTA准确率，并且在音视频事件分类任务上与AudioSet上先前最佳的监督预训练模型相当。代码和预训练模型位于https://github.com/yuangongnd/cav-mae。

英文摘要：

In this paper, we first extend the recent Masked Auto-Encoder (MAE) model from a single modality to audio-visual multi-modalities. Subsequently, we propose the Contrastive Audio-Visual Masked Auto-Encoder (CAV-MAE) by combining contrastive learning and masked data modeling, two major self-supervised learning frameworks, to learn a joint and coordinated audio-visual representation. Our experiments show that the contrastive audio-visual correspondence learning objective not only enables the model to perform audio-visual retrieval tasks, but also helps the model learn a better joint representation. As a result, our fully self-supervised pretrained CAV-MAE achieves a new SOTA accuracy of 65.9% on VGGSound, and is comparable with the previous best supervised pretrained model on AudioSet in the audio-visual event classification task. Code and pretrained models are at https://github.com/yuangongnd/cav-mae.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。