MAViL：掩码音视频学习器
MAViL: Masked Audio-Video Learners
AI总结：
本文提出 MAViL，融合掩码重建、带掩码的模态内/间对比学习及联合特征自训练，学习音视频表示，并在 AudioSet 与 VGGSound 上首次以自监督超越外部监督模型。
摘要：
我们提出了掩码音视频学习器（Masked Audio-Video Learners，MAViL），用于训练音视频表示。我们的方法通过三种互补形式的自监督进行学习：（1）重建被掩码的音频和视频输入数据；（2）结合掩码的模态内与模态间对比学习；（3）通过重建由前两个目标学习到的联合音视频上下文化特征进行自训练。使用 MAViL 进行预训练，不仅使模型在音视频分类和检索任务上表现良好，还提升了每种模态单独的表示，且在微调或推理时无需使用来自另一模态的信息。实验上，MAViL 在 AudioSet（53.1 mAP）和 VGGSound（67.1% 准确率）上创下了新的最优水平。这是自监督音视频模型首次在这些基准上超越使用外部监督的模型。
英文摘要：
We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
