LipSody：通过增强语调一致性进行唇到语音合成
LipSody: Lip-to-Speech Synthesis with Enhanced Prosody Consistency
AI总结：
LipSody通过结合说话人身份、语言内容和情感上下文，提升唇到语音合成的语调一致性。
摘要：
唇到语音合成旨在通过从唇部动作中重建语言内容直接生成语音音频，提供在音频信号不可用或降质时有价值的应用。尽管最近基于扩散的模型如LipVoicer在重建语言内容方面表现出色，但它们通常缺乏语调一致性。在本工作中，我们提出LipSody，一种增强语调一致性的唇到语音框架。LipSody引入了一种语调引导策略，利用三个互补的线索：从面部图像中提取的说话人身份、从唇部动作中获得的语言内容，以及从面部视频中推断出的情感上下文。实验结果表明，与先前方法相比，LipSody显著提高了与语调相关的指标，包括全局和局部音高偏差、能量一致性和说话人相似性。
英文摘要：
Lip-to-speech synthesis aims to generate speech audio directly from silent facial video by reconstructing linguistic content from lip movements, providing valuable applications in situations where audio signals are unavailable or degraded. While recent diffusion-based models such as LipVoicer have demonstrated impressive performance in reconstructing linguistic content, they often lack prosodic consistency. In this work, we propose LipSody, a lip-to-speech framework enhanced for prosody consistency. LipSody introduces a prosody-guiding strategy that leverages three complementary cues: speaker identity extracted from facial images, linguistic content derived from lip movements, and emotional context inferred from face video. Experimental results demonstrate that LipSody substantially improves prosody-related metrics, including global and local pitch deviations, energy consistency, and speaker similarity, compared to prior approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
