arXiv 2409.093982025-03-21eess.AScs.SD

无并行训练数据的语言查询目标声音提取

Language-Queried Target Sound Extraction Without Parallel Training Data

School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）；North Carolina State University（北卡罗来纳州立大学）；ARC Lab, Tencent PCG（腾讯PCG ARC实验室）；Key Laboratory of System and Control, AMSS, Chinese Academy of Sciences（中国科学院数学与系统科学研究院系统与控制重点实验室）；The Chinese University of Hong Kong（香港中文大学）

Hao Ma, Zhiyuan Peng, Xu Li, Yukai Li, Mingjie Shao, Qiuqiang Kong, Ju Liu

备注：: Accepted by ICASSP 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对语言查询目标声音提取需大量并行标注数据的问题，提出基于CLAP的无并行数据训练方案，通过检索增强策略解决模态差距与过拟合问题，性能优于现有最优方法且泛化性更好。

摘要：

语言查询目标声音提取（TSE）旨在根据语言查询从混合声音中提取特定声音。传统的全监督训练方案需要大量标注的并行音频-文本数据，这一过程耗费大量人力。我们提出了一种无需并行数据的训练方案，通过利用对比语言-音频预训练模型（CLAP），仅需未标注的音频片段即可完成TSE模型训练。在基础的无并行数据训练阶段，使用预训练的CLAP音频编码器对目标音频进行编码以生成条件嵌入，而在测试阶段，用户的语言查询则由CLAP文本编码器编码作为条件嵌入。这种基础方法假设文本与音频嵌入之间完全对齐，而这一假设并不符合实际。训练与测试的不匹配带来了两大主要挑战：一是文本与音频之间持续存在的模态差距，二是训练过程中目标音频嵌入暴露丰富声学细节导致的过拟合风险。为解决这一问题，我们提出了检索增强策略。具体而言，我们利用大语言模型（LLM）生成的音频字幕构建嵌入缓存。训练时，目标音频嵌入从该缓存中检索文本嵌入作为条件嵌入，确保训练与测试的模态一致并消除信息泄露。大量实验结果表明，我们的检索增强方法相比现有最优方法实现了持续且显著的性能提升，具备更好的泛化能力。

英文摘要：

Language-queried target sound extraction (TSE) aims to extract specific sounds from mixtures based on language queries. Traditional fully-supervised training schemes require extensively annotated parallel audio-text data, which are labor-intensive. We introduce a parallel-data-free training scheme, requiring only unlabelled audio clips for TSE model training by utilizing the contrastive language-audio pre-trained model (CLAP). In a vanilla parallel-data-free training stage, target audio is encoded using the pre-trained CLAP audio encoder to form a condition embedding, while during testing, user language queries are encoded by CLAP text encoder as the condition embedding. This vanilla approach assumes perfect alignment between text and audio embeddings, which is unrealistic. Two major challenges arise from training-testing mismatch: the persistent modality gap between text and audio and the risk of overfitting due to the exposure of rich acoustic details in target audio embedding during training. To address this, we propose a retrieval-augmented strategy. Specifically, we create an embedding cache using audio captions generated by a large language model (LLM). During training, target audio embeddings retrieve text embeddings from this cache to use as condition embeddings, ensuring consistent modalities between training and testing and eliminating information leakage. Extensive experiment results show that our retrieval-augmented approach achieves consistent and notable performance improvements over existing state-of-the-art with better generalizability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。