arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.12807cs.SDeess.AS
收藏

通用音频表示的多模态自监督学习

Multimodal Self-Supervised Learning of General Audio Representations

DeepMind(深度思维)

Luyu Wang, Pauline Luc, Adria Recasens, Jean-Baptiste Alayrac, Aaron van den Oord

摘要:

我们提出了一个多模态框架,从视频中学习一般的音频表示。现有的对比音频表示学习方法主要集中在训练过程中单独使用音频模态。在这项工作中,我们表明,视频中包含的额外信息可以用来大大提高学习功能。首先,我们证明了我们的对比框架不需要高分辨率的图像来学习良好的音频特征。这使我们能够扩大训练批次的大小,同时将额外视频模态引起的计算负载保持在合理的水平。其次,我们使用混合不同样本的增强。我们表明,这是有效的,使代理任务更难,从而导致大量的性能提高时,增加批量大小。因此,我们的音频模型在AudioSet分类下游任务上实现了最先进的42.4 mAP,缩小了在同一数据集上训练的监督和自监督方法之间的差距。此外,我们表明,我们的方法在广泛的非语义音频任务,包括说话人辨识,关键字定位,语言识别和乐器分类是有利的。

英文摘要:

We present a multimodal framework to learn general audio representations from videos. Existing contrastive audio representation learning methods mainly focus on using the audio modality alone during training. In this work, we show that additional information contained in video can be utilized to greatly improve the learned features. First, we demonstrate that our contrastive framework does not require high resolution images to learn good audio features. This allows us to scale up the training batch size, while keeping the computational load incurred by the additional video modality to a reasonable level. Second, we use augmentations that mix together different samples. We show that this is effective to make the proxy task harder, which leads to substantial performance improvements when increasing the batch size. As a result, our audio model achieves a state-of-the-art of 42.4 mAP on the AudioSet classification downstream task, closing the gap between supervised and self-supervised methods trained on the same dataset. Moreover, we show that our method is advantageous on a broad range of non-semantic audio tasks, including speaker identification, keyword spotting, language identification, and music instrument classification.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑