Face-StyleSpeech：通过改进的人脸到语音映射增强基于人脸图像的零样本语音合成
Face-StyleSpeech: Enhancing Zero-shot Speech Synthesis from Face Images with Improved Face-to-Speech Mapping
AI总结：
本文提出Face-StyleSpeech零样本TTS模型，通过分离人脸编码器与韵律编码器，仅从人脸图像生成自然语音，实验证明其效果优于基线方法且适用于未见过的面孔。
摘要：
从人脸图像生成语音对于开发能够使用其独特声音进行交互的虚拟人至关重要，而无需依赖预先录制的人类语音。在本文中，我们提出了Face-StyleSpeech，这是一种零样本文本到语音（TTS）合成模型，它以人脸图像而非参考语音为条件生成自然语音。我们假设从人脸图像中学习完整的韵律特征面临重大挑战。为了解决这一问题，我们的TTS模型同时引入了人脸编码器和韵律编码器。韵律编码器专门用于建模人脸图像无法完全捕捉的语音风格特征，从而使人脸编码器能够专注于提取说话人特定特征，如音色。实验结果表明，Face-StyleSpeech能够比基线方法更有效地从人脸图像生成更自然的语音，即使对于未见过的面孔也是如此。样本可在我们的演示页面上获取。
英文摘要：
Generating speech from a face image is crucial for developing virtual humans capable of interacting using their unique voices, without relying on pre-recorded human speech. In this paper, we propose Face-StyleSpeech, a zero-shot Text-To-Speech (TTS) synthesis model that generates natural speech conditioned on a face image rather than reference speech. We hypothesize that learning entire prosodic features from a face image poses a significant challenge. To address this, our TTS model incorporates both face and prosody encoders. The prosody encoder is specifically designed to model speech style characteristics that are not fully captured by the face image, allowing the face encoder to focus on extracting speaker-specific features such as timbre. Experimental results demonstrate that Face-StyleSpeech effectively generates more natural speech from a face image than baselines, even for unseen faces. Samples are available on our demo page.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
