Text2Lip：通过视素引导渲染从文本生成渐进式唇形同步说话人脸
Text2Lip: Progressive Lip-Synced Talking Face Generation from Text via Viseme-Guided Rendering
AI总结：
提出Text2Lip，一种基于视素引导的文本驱动说话人脸生成框架，通过课程学习渐进替换音频并用地标引导渲染，实现高语义保真和鲁棒生成。
摘要：
生成语义连贯且视觉准确的说话人脸需要弥合语言意义与面部发音之间的鸿沟。尽管音频驱动的方法仍然普遍，但它们对高质量配对音视频数据的依赖以及将声学映射到唇部运动的内在模糊性，在可扩展性和鲁棒性方面构成了重大挑战。为解决这些问题，我们提出了Text2Lip，一个以视素为中心的框架，通过将文本输入嵌入到结构化的视素序列中，构建了一个可解释的语音-视觉桥梁。这些中层单元作为唇部运动预测的语言学基础先验。此外，我们设计了一种基于课程学习的渐进式视素-音频替换策略，使模型能够通过跨模态注意力逐渐从真实音频过渡到从增强视素特征重建的伪音频。这允许在存在音频和无音频场景下均实现鲁棒生成。最后，一个地标引导的渲染器合成具有准确唇部同步的照片级逼真面部视频。广泛评估表明，Text2Lip在语义保真度、视觉真实感和模态鲁棒性方面优于现有方法，为可控且灵活的说话人脸生成建立了新范式。我们的项目主页是https://plyon1.github.io/Text2Lip/。
英文摘要：
Generating semantically coherent and visually accurate talking faces requires bridging the gap between linguistic meaning and facial articulation. Although audio-driven methods remain prevalent, their reliance on high-quality paired audio visual data and the inherent ambiguity in mapping acoustics to lip motion pose significant challenges in terms of scalability and robustness. To address these issues, we propose Text2Lip, a viseme-centric framework that constructs an interpretable phonetic-visual bridge by embedding textual input into structured viseme sequences. These mid-level units serve as a linguistically grounded prior for lip motion prediction. Furthermore, we design a progressive viseme-audio replacement strategy based on curriculum learning, enabling the model to gradually transition from real audio to pseudo-audio reconstructed from enhanced viseme features via cross-modal attention. This allows for robust generation in both audio-present and audio-free scenarios. Finally, a landmark-guided renderer synthesizes photorealistic facial videos with accurate lip synchronization. Extensive evaluations show that Text2Lip outperforms existing approaches in semantic fidelity, visual realism, and modality robustness, establishing a new paradigm for controllable and flexible talking face generation. Our project homepage is https://plyon1.github.io/Text2Lip/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
