Diffusion Models 作为 Masked Audio-Video Learners
Diffusion Models as Masked Audio-Video Learners
AI总结：
研究扩散模型与MAViL框架的协同作用，通过引入扩散模型及掩码比率课程、自适应批量大小等训练效率方法，在保持下游音频分类任务性能的同时，将预训练FLOPS减少32%、挂钟时间减少18%。
摘要：
在过去的几年中，音频和视觉信号之间的同步已被利用来学习更丰富的音视频表示。得益于大量可用的无标签视频，许多无监督训练框架在各种下游音频和视频任务中展示了令人印象深刻的结果。最近，Masked Audio-Video Learners (MAViL) 已成为最先进的音视频预训练框架。MAViL 将对比学习与掩码自编码相结合，通过融合来自两种模态的信息来联合重建音频频谱图和视频帧。在本文中，我们研究了扩散模型与 MAViL 之间的潜在协同作用，寻求从这两个框架中获得互惠互利。将扩散引入 MAViL，结合包括利用掩码比率课程和自适应批量大小在内的各种训练效率方法，使得预训练浮点运算次数 (FLOPS) 显著减少 32%，预训练挂钟时间减少 18%。至关重要的是，与 MAViL 的性能相比，这种提升的效率并未损害模型在下游音频分类任务中的性能。
英文摘要：
Over the past several years, the synchronization between audio and visual signals has been leveraged to learn richer audio-visual representations. Aided by the large availability of unlabeled videos, many unsupervised training frameworks have demonstrated impressive results in various downstream audio and video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. MAViL couples contrastive learning with masked autoencoding to jointly reconstruct audio spectrograms and video frames by fusing information from both modalities. In this paper, we study the potential synergy between diffusion models and MAViL, seeking to derive mutual benefits from these two frameworks. The incorporation of diffusion into MAViL, combined with various training efficiency methodologies that include the utilization of a masking ratio curriculum and adaptive batch sizing, results in a notable 32% reduction in pre-training Floating-Point Operations (FLOPS) and an 18% decrease in pre-training wall clock time. Crucially, this enhanced efficiency does not compromise the model's performance in downstream audio-classification tasks when compared to MAViL's performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
