CapTalk: 文本引导的风格化与语音驱动的3D头部动画
CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation
AI总结：
提出CapTalk框架，通过文本描述控制说话风格和情感，结合语音驱动生成同步唇动和面部表情，支持动态情感变化。
摘要：
音频驱动的3D面部动画旨在从任意音频片段生成同步的唇部运动和生动的面部表情。现有方法虽能产生同步唇动，但通常依赖预定义的身份或风格潜在特征，限制了用户自由控制说话风格的能力。此外，将固定风格或身份应用于整个音频片段通常导致面部动画风格无法适应音频的情感内容。为解决这些挑战，我们重新审视风格与情感的纠缠，构建了一个包含风格和情感文本描述的大规模数据集，并提出了一种新颖的说话头生成框架，能够分别控制风格和情感。我们的模型以说话风格和角色情感的文本描述以及驱动音频流为输入，能够实时生成与描述高度同步的唇部运动和面部表情。此外，我们的模型在推理时支持动态情感控制，能够处理目标情感在语音过程中变化的情况。
英文摘要：
Audio-driven 3D facial animation aims to generate synchronized lip movements and vivid facial expressions from arbitrary audio clips. While existing methods can produce synchronized lip motions, they often rely on predefined identity or style latent features, which limits users' ability to freely control speaking styles. Moreover, applying a fixed style or identity to an entire audio segment typically results in facial animation styles that do not adapt to the emotional content of the audio. To address these challenges, we revisit the entanglement between style and emotion, construct a large-scale dataset with textual descriptions of both style and emotion, and propose a novel talking head generation framework that enables separate control over style and emotion. Our model takes as input both textual descriptions of speaking style and character emotion, as well as the driving audio stream, enabling real-time generation of highly synchronized lip movements and facial expressions that match the provided descriptions. Furthermore, our model supports dynamic emotion control during inference, allowing it to handle scenarios where the target emotion changes throughout the speech.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
