分离你描述的任何声音
Separate Anything You Describe
AI总结:
本文提出语言查询的开放域音频源分离基础模型AudioSep,经大规模多模态训练,在多任务上展现强零样本分离性能,并开源代码、基准与预训练模型。
摘要:
语言查询音频源分离(LASS)是计算听觉场景分析(CASA)的一种新范式。LASS旨在根据自然语言查询从音频混合中分离目标声音,从而为数字音频应用提供自然且可扩展的接口。近期关于LASS的研究尽管在特定声源(例如乐器、有限类别的音频事件)上取得了令人鼓舞的分离性能,却无法在开放域中分离各类音频概念。在本工作中,我们提出AudioSep,这是一个利用自然语言查询进行开放域音频源分离的基础模型。我们在大规模多模态数据集上训练AudioSep,并在音频事件分离、乐器分离和语音增强等众多任务上广泛评估其能力。AudioSep以音频字幕或文本标签作为查询,展现出强大的分离性能和出色的零样本泛化能力,显著优于以往以音频查询和语言查询的声音分离模型。为保证本工作的可复现性,我们将在https://github.com/Audio-AGI/AudioSep发布源代码、评测基准和预训练模型。
英文摘要:
Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA). LASS aims to separate a target sound from an audio mixture given a natural language query, which provides a natural and scalable interface for digital audio applications. Recent works on LASS, despite attaining promising separation performance on specific sources (e.g., musical instruments, limited classes of audio events), are unable to separate audio concepts in the open domain. In this work, we introduce AudioSep, a foundation model for open-domain audio source separation with natural language queries. We train AudioSep on large-scale multimodal datasets and extensively evaluate its capabilities on numerous tasks including audio event separation, musical instrument separation, and speech enhancement. AudioSep demonstrates strong separation performance and impressive zero-shot generalization ability using audio captions or text labels as queries, substantially outperforming previous audio-queried and language-queried sound separation models. For reproducibility of this work, we will release the source code, evaluation benchmark and pre-trained model at: https://github.com/Audio-AGI/AudioSep.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。