EMOdiffhead：通过扩散模型实现说话人生成中的连续情感控制
EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion
AI总结：
针对现有音频驱动说话人生成方法缺乏情感控制的问题，提出EMOdiffhead方法，结合DECA提取的表情向量与音频引导扩散模型，实现情感的细粒度控制与单次生成，性能达当前最优。
摘要：
音频驱动的人像动画任务旨在利用身份图像和语音音轨生成说话人视频。现有方法大多聚焦于唇形同步和视频质量，却很少有研究应对生成情感驱动的说话人视频这一挑战。控制与编辑情感的能力对于生成富有表现力且逼真的动画至关重要。针对这一挑战，我们提出EMOdiffhead，这是一种用于情感说话人视频生成的新方法，不仅能实现情感类别与强度的细粒度控制，还支持单次生成。鉴于FLAME 3D模型在表情建模上的线性特性，我们利用DECA（详细表情捕获与动画）方法提取表情向量，将其与音频结合，引导扩散模型生成唇形同步精准、情感表现力丰富的视频。该方法既能从与情感无关的数据中学习丰富的面部信息，又能促进情感视频的生成，有效克服了情感数据存在的面部与背景信息多样性不足等局限，也解决了与情感无关的数据中缺乏情感细节的问题。大量实验和用户研究表明，与其他情感人像动画方法相比，我们的方法达到了当前最优性能。
英文摘要：
The task of audio-driven portrait animation involves generating a talking head video using an identity image and an audio track of speech. While many existing approaches focus on lip synchronization and video quality, few tackle the challenge of generating emotion-driven talking head videos. The ability to control and edit emotions is essential for producing expressive and realistic animations. In response to this challenge, we propose EMOdiffhead, a novel method for emotional talking head video generation that not only enables fine-grained control of emotion categories and intensities but also enables one-shot generation. Given the FLAME 3D model's linearity in expression modeling, we utilize the DECA method to extract expression vectors, that are combined with audio to guide a diffusion model in generating videos with precise lip synchronization and rich emotional expressiveness. This approach not only enables the learning of rich facial information from emotion-irrelevant data but also facilitates the generation of emotional videos. It effectively overcomes the limitations of emotional data, such as the lack of diversity in facial and background information, and addresses the absence of emotional details in emotion-irrelevant data. Extensive experiments and user studies demonstrate that our approach achieves state-of-the-art performance compared to other emotion portrait animation methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
