自监督学习是否足以填补空白？语音修复研究
Is Self-Supervised Learning Enough to Fill in the Gap? A Study on Speech Inpainting
AI总结：
本研究探讨了使用自监督学习训练的语音编码器进行语音修复，发现其在单人语音场景中表现更佳，而在多人语音场景中预训练编码器更有效。
摘要：
语音修复是指利用周围上下文来重建受损或缺失的语音段，这一过程与自监督学习（SSL）中用于语音编码器的预设任务非常相似。本研究探讨了使用经过SSL训练的语音编码器进行修复，而无需额外训练，仅通过添加解码器生成波形的方法。我们比较了这种方法与对语音编码器进行监督微调以完成下游任务（即修复）的对比。实际上，我们整合了HuBERT作为SSL编码器和HiFi-GAN作为解码器，在两种配置中进行：（1）微调解码器以与冻结的预训练编码器的输出对齐；（2）基于冻结解码器输入对编码器进行修复任务的微调。评估在单人和多人语音条件下使用领域内和领域外数据集进行，包括未见过的说话者、多样的说话风格和噪声。考虑了有信息和无信息修复场景，其中损坏段的位置是已知或未知。所提出的基于SSL的方法与多个基线方法进行了基准测试，包括结合自动语音识别和零样本文本到语音合成的文本引导方法。性能通过客观指标和感知评估来评估。结果表明，两种方法均优于基线，成功重建语音段长达200毫秒，有时甚至达到400毫秒。值得注意的是，微调SSL编码器在单人语音设置中实现了更准确的语音重建，而预训练编码器在多人语音场景中证明更有效。这表明SSL预设任务可以转移到语音修复中，使预训练编码器能够实现成功的语音重建。
英文摘要：
Speech inpainting consists in reconstructing corrupted or missing speech segments using surrounding context, a process that closely resembles the pretext tasks in Self-Supervised Learning (SSL) for speech encoders. This study investigates using SSL-trained speech encoders for inpainting without any additional training beyond the initial pretext task, and simply adding a decoder to generate a waveform. We compare this approach to supervised fine-tuning of speech encoders for a downstream task -- here, inpainting. Practically, we integrate HuBERT as the SSL encoder and HiFi-GAN as the decoder in two configurations: (1) fine-tuning the decoder to align with the frozen pre-trained encoder's output and (2) fine-tuning the encoder for an inpainting task based on a frozen decoder's input. Evaluations are conducted under single- and multi-speaker conditions using in-domain datasets and out-of-domain datasets (including unseen speakers, diverse speaking styles, and noise). Both informed and blind inpainting scenarios are considered, where the position of the corrupted segment is either known or unknown. The proposed SSL-based methods are benchmarked against several baselines, including a text-informed method combining automatic speech recognition with zero-shot text-to-speech synthesis. Performance is assessed using objective metrics and perceptual evaluations. The results demonstrate that both approaches outperform baselines, successfully reconstructing speech segments up to 200 ms, and sometimes up to 400 ms. Notably, fine-tuning the SSL encoder achieves more accurate speech reconstruction in single-speaker settings, while a pre-trained encoder proves more effective for multi-speaker scenarios. This demonstrates that an SSL pretext task can transfer to speech inpainting, enabling successful speech reconstruction with a pre-trained encoder.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
