arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.13199eess.AScs.SD
收藏

基于预训练自监督学习模型的目标语音提取

Target Speech Extraction with Pre-trained Self-supervised Learning Models

Brno University of Technology(布尔诺理工大学);NTT Corporation(NTT公司)

Junyi Peng, Marc Delcroix, Tsubasa Ochiai, Oldrich Plchot, Shoko Araki, Jan Cernocky

AI总结:

本文研究如何利用预训练自监督学习模型提升目标语音提取性能,提出自适应输入增强器和说话人编码器两个SSL模块,在LibriMix上取得14.0 dB的SI-SDR改进。

摘要:

预训练的自监督学习(SSL)模型已在多种语音任务中取得显著成功,但其在目标语音提取(TSE)中的潜力尚未被充分挖掘。TSE旨在根据注册话语从混合语音中提取目标说话人的语音。我们在TSE框架内利用预训练SSL模型实现两个目的:处理输入混合语音,以及从注册话语中推导说话人嵌入。本文重点研究如何有效利用SSL模型进行TSE。我们首先按照SUPERB原则引入一个新的TSE下游任务。这一简单实验展示了SSL模型用于TSE的潜力,但提取性能仍远落后于最先进水平。随后,我们通过引入两个基于SSL的模块来扩展一个强大的TSE架构:自适应输入增强器(AIE)和说话人编码器。具体而言,所提出的AIE通过渐进式上采样调整CNN编码器和Transformer块的时间分辨率,利用CNN编码器的中间表示,同时捕获细粒度和层次化特征。我们的方法优于当前的TSE系统,在LibriMix上实现了14.0 dB的SI-SDR改进。此外,通过微调包括SSL模型参数在内的整个模型,我们可以进一步将性能提升0.7 dB。

英文摘要:

Pre-trained self-supervised learning (SSL) models have achieved remarkable success in various speech tasks. However, their potential in target speech extraction (TSE) has not been fully exploited. TSE aims to extract the speech of a target speaker in a mixture guided by enrollment utterances. We exploit pre-trained SSL models for two purposes within a TSE framework, i.e., to process the input mixture and to derive speaker embeddings from the enrollment. In this paper, we focus on how to effectively use SSL models for TSE. We first introduce a novel TSE downstream task following the SUPERB principles. This simple experiment shows the potential of SSL models for TSE, but extraction performance remains far behind the state-of-the-art. We then extend a powerful TSE architecture by incorporating two SSL-based modules: an Adaptive Input Enhancer (AIE) and a speaker encoder. Specifically, the proposed AIE utilizes intermediate representations from the CNN encoder by adjusting the time resolution of CNN encoder and transformer blocks through progressive upsampling, capturing both fine-grained and hierarchical features. Our method outperforms current TSE systems achieving a SI-SDR improvement of 14.0 dB on LibriMix. Moreover, we can further improve performance by 0.7 dB by fine-tuning the whole model including the SSL model parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑