arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.15147eess.AScs.AIcs.CLcs.SDeess.SP
收藏

分离您所描述的:存储查询音频源分离

Separate What You Describe: Language-Queried Audio Source Separation

University of Surrey(萨里大学);ByteDance(字节跳动)

Xubo Liu, Haohe Liu, Qiuqiang Kong, Xinhao Mei, Jinzheng Zhao, Qiushi Huang, Mark D. Plumbley, Wenwu Wang

摘要:

在本文中,我们介绍了语言查询音频源分离(LASS)的任务,其目的是基于目标源的自然语言查询(例如,“一个人讲了一个笑话,人们跟着笑”)。LASS中的一个独特挑战与自然语言描述的复杂性及其与音频源的关系有关。为了解决这个问题,我们提出了LASS-Net,这是一种端到端的神经网络,可以学习联合处理声学和语言信息,并从音频混合中分离出与语言查询一致的目标源。我们评估我们提出的系统与AudioCaps数据集创建的数据集的性能。实验结果表明,LASS-Net实现了相当大的改进基线方法。此外,我们观察到LASS-Net在使用不同的人类注释描述作为查询时取得了有希望的泛化结果,表明其在现实世界中的潜在用途。分离的音频样本和源代码可在https://liuxubo717.github.io/LASS-demopage上获得。

英文摘要:

In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language query of the target source (e.g., "a man tells a joke followed by people laughing"). A unique challenge in LASS is associated with the complexity of natural language description and its relation with the audio sources. To address this issue, we proposed LASS-Net, an end-to-end neural network that is learned to jointly process acoustic and linguistic information, and separate the target source that is consistent with the language query from an audio mixture. We evaluate the performance of our proposed system with a dataset created from the AudioCaps dataset. Experimental results show that LASS-Net achieves considerable improvements over baseline methods. Furthermore, we observe that LASS-Net achieves promising generalization results when using diverse human-annotated descriptions as queries, indicating its potential use in real-world scenarios. The separated audio samples and source code are available at https://liuxubo717.github.io/LASS-demopage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑