arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.10052cs.SDeess.AS
收藏

基于条件隐扩散的双上下文学习语音增强方法

Conditional Latent Diffusion-Based Speech Enhancement Via Dual Context Learning

Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)

Shengkui Zhao, Zexu Pan, Kun Zhou, Yukun Ma, Chong Zhang, Bin Ma

AI总结:

针对现有扩散语音增强方法生成复杂度高、噪声建模不足的问题,提出结合条件隐扩散与双上下文学习的方法,通过低维隐空间运算降复杂度,在更少迭代下实现更优性能与域外泛化能力。

摘要:

近年来,扩散概率模型的应用通过生成式方法推动了语音增强技术的发展。然而,现有基于扩散的方法聚焦于高维波形或频谱域的生成过程,导致生成复杂度升高、推理速度变慢。此外,这类方法主要对干净语音分布进行建模,对噪声分布的探索有限,从而制约了扩散模型在语音增强任务中的判别能力。为解决上述问题,我们提出一种将条件隐扩散模型(conditional latent diffusion model, cLDM)与双上下文学习(dual-context learning, DCL)相结合的新方法。该方法利用变分自编码器(variational autoencoder, VAE)将梅尔频谱压缩至低维隐空间;随后通过DCL过程,用cLDM将干净语音与背景噪声的隐表示均转换为高斯噪声,并训练一个参数化模型,以含噪隐表示和文本嵌入为条件逆转该过程。通过在低维空间中运算,隐表示降低了生成过程的复杂度,而DCL过程提升了模型应对多样及未见噪声环境的能力。实验结果表明,与现有基于扩散的方法相比,所提方法即便使用更少的迭代步骤仍表现出优异性能,且模型在域外噪声数据集上具备更出色的泛化能力(代码见https://github.com/modelscope/ClearerVoice-Studio)。

英文摘要:

Recently, the application of diffusion probabilistic models has advanced speech enhancement through generative approaches. However, existing diffusion-based methods have focused on the generation process in high-dimensional waveform or spectral domains, leading to increased generation complexity and slower inference speeds. Additionally, these methods have primarily modelled clean speech distributions, with limited exploration of noise distributions, thereby constraining the discriminative capability of diffusion models for speech enhancement. To address these issues, we propose a novel approach that integrates a conditional latent diffusion model (cLDM) with dual-context learning (DCL). Our method utilizes a variational autoencoder (VAE) to compress mel-spectrograms into a low-dimensional latent space. We then apply cLDM to transform the latent representations of both clean speech and background noise into Gaussian noise by the DCL process, and a parameterized model is trained to reverse this process, conditioned on noisy latent representations and text embeddings. By operating in a lower-dimensional space, the latent representations reduce the complexity of the generation process, while the DCL process enhances the model's ability to handle diverse and unseen noise environments. Our experiments demonstrate the strong performance of the proposed approach compared to existing diffusion-based methods, even with fewer iterative steps, and highlight the superior generalization capability of our models to out-of-domain noise datasets (https://github.com/modelscope/ClearerVoice-Studio).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑