SC VALL-E：风格可控的零样本文本转语音合成器
SC VALL-E: Style-Controllable Zero-Shot Text to Speech Synthesizer
AI总结：
本研究基于VALL-E提出风格可控的SC VALL-E模型，通过风格网络控制语音属性，在零样本场景下生成多样表现力语音，性能优于现有代表性模型。
摘要：
为了控制语音的多种特征并生成所需的音色，富有表现力的语音合成模型是通过在数据集中添加包含不同说话人、多种情绪和不同说话风格的语料库来训练的。在本文中，我们基于神经编解码器语言模型（称为VALL-E）提出了一种风格控制（style control，SC）VALL-E模型，该模型遵循生成式预训练Transformer 3（GPT-3）的结构。所提出的SC VALL-E接收文本句子和提示音频作为输入，其设计目的不是简单地模仿提示音频的特征，而是通过控制属性来生成多样化的语音，从而实现可控语音生成。我们在新设计的风格网络的风格嵌入矩阵中识别出代表情绪、说话速率、音高和语音强度等属性的token，并设计了一个可以控制这些属性的模型。为了评估SC VALL-E的性能，我们与三个具有代表性的表现力语音合成模型进行了对比实验：全局风格token（global style token，GST）Tacotron2、变分自编码器（variational autoencoder，VAE）Tacotron2以及原始VALL-E。我们采用词错误率（word error rate，WER）、F0浊音误差（F0 voiced error，FVE）和F0总音高误差（F0 gross pitch error，F0GPE）作为评估指标，来衡量生成句子的准确性。为了比较合成语音的质量，我们测量了比较平均意见得分（comparative mean option score，CMOS）和相似度平均意见得分（similarity mean option score，SMOS）。为了评估生成语音的风格控制能力，我们通过修改训练后的token来观察F0和梅尔频谱图的变化。当使用训练数据中不存在的提示音频时，SC VALL-E能够生成多种富有表现力的语音，并且与现有模型相比表现出具有竞争力的性能。我们的实现代码、预训练模型和音频样本已发布在GitHub上。
英文摘要：
Expressive speech synthesis models are trained by adding corpora with diverse speakers, various emotions, and different speaking styles to the dataset, in order to control various characteristics of speech and generate the desired voice. In this paper, we propose a style control (SC) VALL-E model based on the neural codec language model (called VALL-E), which follows the structure of the generative pretrained transformer 3 (GPT-3). The proposed SC VALL-E takes input from text sentences and prompt audio and is designed to generate controllable speech by not simply mimicking the characteristics of the prompt audio but by controlling the attributes to produce diverse voices. We identify tokens in the style embedding matrix of the newly designed style network that represent attributes such as emotion, speaking rate, pitch, and voice intensity, and design a model that can control these attributes. To evaluate the performance of SC VALL-E, we conduct comparative experiments with three representative expressive speech synthesis models: global style token (GST) Tacotron2, variational autoencoder (VAE) Tacotron2, and original VALL-E. We measure word error rate (WER), F0 voiced error (FVE), and F0 gross pitch error (F0GPE) as evaluation metrics to assess the accuracy of generated sentences. For comparing the quality of synthesized speech, we measure comparative mean option score (CMOS) and similarity mean option score (SMOS). To evaluate the style control ability of the generated speech, we observe the changes in F0 and mel-spectrogram by modifying the trained tokens. When using prompt audio that is not present in the training data, SC VALL-E generates a variety of expressive sounds and demonstrates competitive performance compared to the existing models. Our implementation, pretrained models, and audio samples are located on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
