MEDTalk：基于解耦嵌入的动态情感多模态可控3D面部动画
MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding
AI总结：
提出MEDTalk框架，通过解耦内容与情感嵌入空间实现独立控制，结合多模态输入生成细粒度动态情感的3D面部动画，并可便捷集成至工业生产流程。
摘要：
音频驱动的情感3D面部动画旨在生成同步的唇部动作和生动的面部表情。然而，现有大多数方法关注静态和预定义的情感标签，限制了其多样性和自然性。为应对这些挑战，我们提出MEDTalk，一个用于细粒度和动态情感说话头生成的新颖框架。该方法首先通过精心设计的交叉重构过程，从运动序列中解耦内容和情感嵌入空间，实现对唇部动作和面部表情的独立控制。除常规的音频驱动唇部同步外，我们整合音频和语音文本，预测逐帧强度变化并动态调整静态情感特征，以生成逼真的情感表情。此外，为增强控制和个性化，我们引入多模态输入——包括文本描述和参考表情图像——以指导生成用户指定的面部表情。以MetaHuman为优先，我们的生成结果可便捷地集成到工业生产流程中。代码可在https://github.com/SJTU-Lucy/MEDTalk获取。
英文摘要：
Audio-driven emotional 3D facial animation aims to generate synchronized lip movements and vivid facial expressions. However, most existing approaches focus on static and predefined emotion labels, limiting their diversity and naturalness. To address these challenges, we propose MEDTalk, a novel framework for fine-grained and dynamic emotional talking head generation. Our approach first disentangles content and emotion embedding spaces from motion sequences using a carefully designed cross-reconstruction process, enabling independent control over lip movements and facial expressions. Beyond conventional audio-driven lip synchronization, we integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions. Furthermore, to enhance control and personalization, we incorporate multimodal inputs-including text descriptions and reference expression images-to guide the generation of user-specified facial expressions. With MetaHuman as the priority, our generated results can be conveniently integrated into the industrial production pipeline. The code is available at: https://github.com/SJTU-Lucy/MEDTalk.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
