Vevo：基于自监督解纠缠的可控零样本声音模仿
Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement
AI总结：
Vevo提出一种自监督解纠缠的零样本声音模仿框架，通过两阶段建模实现音色与风格的可控生成，在口音和情感转换任务中达到或超越现有方法。
摘要：
针对特定语音属性（如音色和说话风格）的声音模仿在语音生成中至关重要。然而，现有方法严重依赖标注数据，并且在有效解纠缠音色和风格方面存在困难，导致在实现可控生成（尤其是在零样本场景下）面临挑战。为解决这些问题，我们提出了Vevo，一个具有可控音色和风格的多功能零样本声音模仿框架。Vevo包含两个核心阶段：（1）内容-风格建模：给定文本或语音的内容标记作为输入，我们利用自回归Transformer生成内容-风格标记，该过程由风格参考提示；（2）声学建模：给定内容-风格标记作为输入，我们采用流匹配Transformer生成声学表示，该过程由音色参考提示。为获取语音的内容和内容-风格标记，我们设计了一种完全自监督的方法，逐步解纠缠语音的音色、风格和语言内容。具体而言，我们采用VQ-VAE作为HuBERT连续隐藏特征的标记器。我们将VQ-VAE码本的词汇表大小视为信息瓶颈，并仔细调整以获得解纠缠的语音表示。仅在60K小时的音频书语音数据上进行自监督训练，未对风格特定语料库进行任何微调，Vevo在口音和情感转换任务中达到或超越了现有方法。此外，Vevo在零样本语音转换和文本到语音任务中的有效性进一步展示了其强大的泛化能力和多功能性。音频样本可在https://versavoice.github.io获取。
英文摘要：
The imitation of voice, targeted on specific speech attributes such as timbre and speaking style, is crucial in speech generation. However, existing methods rely heavily on annotated data, and struggle with effectively disentangling timbre and style, leading to challenges in achieving controllable generation, especially in zero-shot scenarios. To address these issues, we propose Vevo, a versatile zero-shot voice imitation framework with controllable timbre and style. Vevo operates in two core stages: (1) Content-Style Modeling: Given either text or speech's content tokens as input, we utilize an autoregressive transformer to generate the content-style tokens, which is prompted by a style reference; (2) Acoustic Modeling: Given the content-style tokens as input, we employ a flow-matching transformer to produce acoustic representations, which is prompted by a timbre reference. To obtain the content and content-style tokens of speech, we design a fully self-supervised approach that progressively decouples the timbre, style, and linguistic content of speech. Specifically, we adopt VQ-VAE as the tokenizer for the continuous hidden features of HuBERT. We treat the vocabulary size of the VQ-VAE codebook as the information bottleneck, and adjust it carefully to obtain the disentangled speech representations. Solely self-supervised trained on 60K hours of audiobook speech data, without any fine-tuning on style-specific corpora, Vevo matches or surpasses existing methods in accent and emotion conversion tasks. Additionally, Vevo's effectiveness in zero-shot voice conversion and text-to-speech tasks further demonstrates its strong generalization and versatility. Audio samples are available at https://versavoice.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
