CLAPSep：利用对比预训练模型实现多模态查询条件下的目标声音提取
CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction
AI总结：
针对现有从零训练的语言查询式目标声音提取方法数据与算力需求高的问题，提出整合对比预训练CLAP模型的CLAPSep方法，支持多模态正负提示，在5个数据集上实现更优性能与泛化性。
摘要：
通用声音分离（USS）旨在从真实世界录音中提取任意类型的声音。这一目标可通过语言查询式目标声音提取（TSE）实现，该方法通常包含两个组件：将用户查询转换为条件嵌入的查询网络，以及据此提取目标声音的分离网络。现有方法通常从零开始训练模型，因此需要大量数据和计算资源，才能让随机初始化的模型理解声音事件并执行相应的分离操作。本文提出将预训练模型整合到TSE模型中以解决上述问题。具体而言，我们对功能强大的对比语言-音频预训练模型（CLAP）进行定制和适配以用于USS，将其命名为CLAPSep。CLAPSep还支持灵活的用户输入，可接收单模态和/或多模态的正向及负向用户提示来执行目标声音提取。CLAPSep的这些关键特性不仅能提升提取性能，还能提高其应用的通用性。我们在5个不同的数据集上开展了大量实验，结果表明所提出的CLAPSep性能优异，具备零样本和少样本泛化能力，且训练收敛速度快，大幅超越了以往的方法。完整代码和部分音频示例已发布，以供复现和评估使用。
英文摘要：
Universal sound separation (USS) aims to extract arbitrary types of sounds from real-world recordings. This can be achieved by language-queried target sound extraction (TSE), which typically consists of two components: a query network that converts user queries into conditional embeddings, and a separation network that extracts the target sound accordingly. Existing methods commonly train models from scratch. As a consequence, substantial data and computational resources are required to make the randomly initialized model comprehend sound events and perform separation accordingly. In this paper, we propose to integrate pre-trained models into TSE models to address the above issue. To be specific, we tailor and adapt the powerful contrastive language-audio pre-trained model (CLAP) for USS, denoted as CLAPSep. CLAPSep also accepts flexible user inputs, taking both positive and negative user prompts of uni- and/or multi-modalities for target sound extraction. These key features of CLAPSep can not only enhance the extraction performance but also improve the versatility of its application. We provide extensive experiments on 5 diverse datasets to demonstrate the superior performance and zero- and few-shot generalizability of our proposed CLAPSep with fast training convergence, surpassing previous methods by a significant margin. Full codes and some audio examples are released for reproduction and evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
