Ada-TTA：迈向自适应高质量的文本到说话人虚拟人合成
Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis
AI总结：
提出Ada-TTA方法，通过零样本多说话人TTS模型解耦文本、音色和韵律，并结合神经渲染技术，在低资源条件下生成高保真、唇音同步且保持身份的说话人虚拟人视频。
摘要：
我们关注一项新任务，即低资源文本到说话人虚拟人。给定仅有几分钟长的带音轨的说话人视频作为训练数据，以及任意文本作为驱动输入，我们旨在合成与输入文本对应的高质量说话人肖像视频。该任务在数字人行业有广阔应用前景，但由于两个挑战尚未在技术上实现：（1）传统多说话人文本到语音系统难以模仿域外音频的音色。（2）在有限训练数据下难以渲染高保真且唇音同步的说话人虚拟人。本文提出自适应文本到说话人虚拟人（Ada-TTA），其（1）设计了一个通用的零样本多说话人TTS模型，能很好地解耦文本内容、音色和韵律；（2）结合神经渲染的最新进展，实现逼真的音频驱动说话人脸视频生成。通过这些设计，我们的方法克服了上述两个挑战，实现了生成保持身份的语音和逼真的说话人视频。实验表明，我们的方法能合成逼真、保持身份且音视频同步的说话人虚拟人视频。
英文摘要：
We are interested in a novel task, namely low-resource text-to-talking avatar. Given only a few-minute-long talking person video with the audio track as the training data and arbitrary texts as the driving input, we aim to synthesize high-quality talking portrait videos corresponding to the input text. This task has broad application prospects in the digital human industry but has not been technically achieved yet due to two challenges: (1) It is challenging to mimic the timbre from out-of-domain audio for a traditional multi-speaker Text-to-Speech system. (2) It is hard to render high-fidelity and lip-synchronized talking avatars with limited training data. In this paper, we introduce Adaptive Text-to-Talking Avatar (Ada-TTA), which (1) designs a generic zero-shot multi-speaker TTS model that well disentangles the text content, timbre, and prosody; and (2) embraces recent advances in neural rendering to achieve realistic audio-driven talking face video generation. With these designs, our method overcomes the aforementioned two challenges and achieves to generate identity-preserving speech and realistic talking person video. Experiments demonstrate that our method could synthesize realistic, identity-preserving, and audio-visual synchronized talking avatar videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
