arXiv 2411.167262024-12-16cs.CVcs.AI

EmotiveTalk：通过音频信息解耦与情感视频扩散实现富有表现力的说话人生成

EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion

University of Science and Technology of China（中国科学技术大学）；Imperial College London（帝国理工学院）；iFLYTEK CO.LTD.（科大讯飞股份有限公司）

Haotian Wang, Yuzhe Weng, Yueyan Li, Zilu Guo, Jun Du, Shutong Niu, Jiefeng Ma, Shan He, Xiaoyan Wu, Qiming Hu, Bing Yin, Cong Liu, Qingfeng Liu

备注：: https://emotivetalk.github.io/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出EmotiveTalk框架，通过音频信息解耦与情感视频扩散，实现可控且稳定的高表现力说话人生成，性能优于现有方法。

摘要：

扩散模型已经彻底改变了说话人生成领域，但在长时间生成中仍面临表现力、可控性和稳定性方面的挑战。在本研究中，我们提出了一个EmotiveTalk框架来解决这些问题。首先，为了实现对唇部运动和面部表情生成的更好控制，设计了一种视觉引导的音频信息解耦（V-AID）方法，以生成与唇部运动和表情对齐的基于音频的解耦表示。具体而言，为了实现音频与面部表情表示空间之间的对齐，我们在V-AID中提出了一个基于扩散的协同语音时间扩展（Di-CTE）模块，用于在多源情感条件约束下生成与表情相关的表示。然后，我们提出了一个精心设计的情感说话人扩散（ETHD）主干网络，以高效生成高表现力的说话人视频，其中包含一个表情解耦注入（EDI）模块，用于自动从参考肖像中解耦表情，同时整合目标表情信息，从而实现更具表现力的生成性能。实验结果表明，EmotiveTalk能够生成富有表现力的说话人视频，确保了所承诺的情感可控性和长时间生成的稳定性，与现有方法相比取得了最先进的性能。

英文摘要：

Diffusion models have revolutionized the field of talking head generation, yet still face challenges in expressiveness, controllability, and stability in long-time generation. In this research, we propose an EmotiveTalk framework to address these issues. Firstly, to realize better control over the generation of lip movement and facial expression, a Vision-guided Audio Information Decoupling (V-AID) approach is designed to generate audio-based decoupled representations aligned with lip movements and expression. Specifically, to achieve alignment between audio and facial expression representation spaces, we present a Diffusion-based Co-speech Temporal Expansion (Di-CTE) module within V-AID to generate expression-related representations under multi-source emotion condition constraints. Then we propose a well-designed Emotional Talking Head Diffusion (ETHD) backbone to efficiently generate highly expressive talking head videos, which contains an Expression Decoupling Injection (EDI) module to automatically decouple the expressions from reference portraits while integrating the target expression information, achieving more expressive generation performance. Experimental results show that EmotiveTalk can generate expressive talking head videos, ensuring the promised controllability of emotions and stability during long-time generation, yielding state-of-the-art performance compared to existing methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。