arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.09418eess.AScs.IRcs.SD
收藏

基于自然语言的音频检索:一项基准研究

Audio Retrieval with Natural Language Queries: A Benchmark Study

University of Tübingen(蒂宾根大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);University of Oxford(牛津大学);University of Cambridge(剑桥大学)

A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques, Zeynep Akata, Samuel Albanie

摘要:

这项工作的目标是跨模态的文本音频和音频文本检索,其目标是从一个候选人的池中检索的音频内容,最佳匹配给定的书面描述,反之亦然。文本音频检索使用户能够通过一个直观的界面搜索大型数据库:他们只需发布他们想要听到的声音的自由形式的自然语言描述。为了研究文本音频和音频文本检索的任务,这在现有的文献中受到了有限的关注,我们引入了三个具有挑战性的新基准。我们首先从AudioCaps和Clotho音频字幕数据集构建文本音频和音频文本检索基准。此外,我们还介绍了SoundDescs基准,它由成对的音频和自然语言描述组成,用于各种声音集合,这些声音与AudioCaps和Clotho中的声音是互补的。我们采用这三个基准来建立跨模态文本音频和音频文本检索的基线,在那里我们展示了预训练对不同音频任务的好处。我们希望我们的基准将激发进一步研究音频检索与自由形式的文本查询。所有使用的数据集的代码、音频功能以及SoundDescs数据集都可以在https://github.com/akoepke/audio-retrieval-benchmark上公开获得。

英文摘要:

The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a given written description and vice versa. Text-audio retrieval enables users to search large databases through an intuitive interface: they simply issue free-form natural language descriptions of the sound they would like to hear. To study the tasks of text-audio and audio-text retrieval, which have received limited attention in the existing literature, we introduce three challenging new benchmarks. We first construct text-audio and audio-text retrieval benchmarks from the AudioCaps and Clotho audio captioning datasets. Additionally, we introduce the SoundDescs benchmark, which consists of paired audio and natural language descriptions for a diverse collection of sounds that are complementary to those found in AudioCaps and Clotho. We employ these three benchmarks to establish baselines for cross-modal text-audio and audio-text retrieval, where we demonstrate the benefits of pre-training on diverse audio tasks. We hope that our benchmarks will inspire further research into audio retrieval with free-form text queries. Code, audio features for all datasets used, and the SoundDescs dataset are publicly available at https://github.com/akoepke/audio-retrieval-benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑