利用多尺度声学提示改进基于语言模型的零样本文本转语音合成
Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts
AI总结：
该研究针对零样本TTS中短声学提示难以还原个人说话风格的问题，基于VALL-E引入多尺度风格与音色提示，提升自然度和说话人相似度。
摘要：
零样本文本转语音（TTS）合成旨在无需自适应参数即可克隆任何未见过说话人的声音。通过将语音波形量化为离散声学令牌，并使用语言模型对这些令牌建模，近期基于语言模型的TTS模型仅借助未见过说话人的3秒声学提示，便展现出零样本说话人自适应能力。然而，它们受到声学提示长度的限制，因此难以克隆个人说话风格。本文基于神经编解码语言模型VALL-E，提出一种带有多尺度声学提示的新型零样本TTS模型。文中提出说话人感知文本编码器，用于从由多个句子组成的风格提示中学习音素级的个人说话风格。随后，采用基于VALL-E的声学解码器在帧级对来自音色提示的音色进行建模并生成语音。实验结果表明，所提方法在自然度和说话人相似度方面优于基线，并可通过扩展到更长的风格提示获得更好性能。
英文摘要：
Zero-shot text-to-speech (TTS) synthesis aims to clone any unseen speaker's voice without adaptation parameters. By quantizing speech waveform into discrete acoustic tokens and modeling these tokens with the language model, recent language model-based TTS models show zero-shot speaker adaptation capabilities with only a 3-second acoustic prompt of an unseen speaker. However, they are limited by the length of the acoustic prompt, which makes it difficult to clone personal speaking style. In this paper, we propose a novel zero-shot TTS model with the multi-scale acoustic prompts based on a neural codec language model VALL-E. A speaker-aware text encoder is proposed to learn the personal speaking style at the phoneme-level from the style prompt consisting of multiple sentences. Following that, a VALL-E based acoustic decoder is utilized to model the timbre from the timbre prompt at the frame-level and generate speech. The experimental results show that our proposed method outperforms baselines in terms of naturalness and speaker similarity, and can achieve better performance by scaling out to a longer style prompt.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
