基于捷径流匹配的节奏可控高效零样本语音转换
Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching
AI总结：
本文提出R-VC模型，通过Hubert内容令牌、掩码生成式Transformer时长建模及捷径流匹配扩散Transformer，实现节奏可控且仅需两步采样的高效零样本语音转换，在自然度、可懂度和风格迁移上超越现有方法。
摘要：
零样本语音转换（VC）旨在将源说话人的音色转换为任意未见过的目标说话人音色，同时保留语音内容。大多数先前工作侧重于保留源的韵律，然而细粒度音色信息可能通过韵律泄漏，并且将目标韵律迁移到合成语音中的研究很少。鉴于此，我们提出了R-VC，一种节奏可控且高效的零样本语音转换模型。R-VC采用数据扰动技术，并将源语音离散化为Hubert内容令牌，消除了大量与内容无关的信息。通过利用掩码生成式Transformer进行上下文时长建模，我们的模型使语言内容时长适应所需的目标说话风格，从而促进目标说话人节奏的迁移。此外，R-VC引入了一个强大的扩散Transformer（DiT），在训练期间采用捷径流匹配，不仅以当前噪声水平为条件，还以所需步长为条件，从而在更少的采样步骤中实现高音色相似度和高质量语音生成，甚至仅需两步，从而最大限度地减少延迟。实验结果表明，R-VC在较小的数据集上实现了与最先进VC方法相当的说话人相似度，并在语音自然度、可懂度和风格迁移性能方面超越了它们。
英文摘要：
Zero-Shot Voice Conversion (VC) aims to transform the source speaker's timbre into an arbitrary unseen one while retaining speech content. Most prior work focuses on preserving the source's prosody, while fine-grained timbre information may leak through prosody, and transferring target prosody to synthesized speech is rarely studied. In light of this, we propose R-VC, a rhythm-controllable and efficient zero-shot voice conversion model. R-VC employs data perturbation techniques and discretize source speech into Hubert content tokens, eliminating much content-irrelevant information. By leveraging a Mask Generative Transformer for in-context duration modeling, our model adapts the linguistic content duration to the desired target speaking style, facilitating the transfer of the target speaker's rhythm. Furthermore, R-VC introduces a powerful Diffusion Transformer (DiT) with shortcut flow matching during training, conditioning the network not only on the current noise level but also on the desired step size, enabling high timbre similarity and quality speech generation in fewer sampling steps, even in just two, thus minimizing latency. Experimental results show that R-VC achieves comparable speaker similarity to state-of-the-art VC methods with a smaller dataset, and surpasses them in terms of speech naturalness, intelligibility and style transfer performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
