arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.15537eess.AScs.SD
收藏

基于度量学习的音频文本跨模态检索

On Metric Learning for Audio-Text Cross-Modal Retrieval

University of Surrey(萨里大学)

Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang

摘要:

音频文本检索的目的是在给定另一模态的查询的情况下从候选者池中检索目标音频片段或字幕。解决这种跨模态检索任务是具有挑战性的,因为它不仅需要学习两种模态的鲁棒特征表示,而且还需要捕获这两种模态之间的细粒度对齐。现有的跨模态检索模型大多是通过度量学习目标来优化的,因为它们都试图将数据映射到嵌入空间,在嵌入空间中,相似的数据靠得很近,不相似的数据相距很远。与其他跨模态的检索任务,如图像-文本和视频-文本检索,音频-文本检索仍然是一个未开发的任务。在这项工作中,我们的目标是研究不同的度量学习目标对音频文本检索任务的影响。我们对AudioCaps和Clotho数据集上流行的度量学习目标进行了广泛的评估。我们证明了自监督学习的NT-Xent损失在不同的数据集和训练设置中表现出稳定的性能,并且优于流行的基于三元组的损失。我们的代码可在https://github.com/XinhaoMei/audio-text_retrieval上获得。

英文摘要:

Audio-text retrieval aims at retrieving a target audio clip or caption from a pool of candidates given a query in another modality. Solving such cross-modal retrieval task is challenging because it not only requires learning robust feature representations for both modalities, but also requires capturing the fine-grained alignment between these two modalities. Existing cross-modal retrieval models are mostly optimized by metric learning objectives as both of them attempt to map data to an embedding space, where similar data are close together and dissimilar data are far apart. Unlike other cross-modal retrieval tasks such as image-text and video-text retrievals, audio-text retrieval is still an unexplored task. In this work, we aim to study the impact of different metric learning objectives on the audio-text retrieval task. We present an extensive evaluation of popular metric learning objectives on the AudioCaps and Clotho datasets. We demonstrate that NT-Xent loss adapted from self-supervised learning shows stable performance across different datasets and training settings, and outperforms the popular triplet-based losses. Our code is available at https://github.com/XinhaoMei/audio-text_retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑