EMO：让表情肖像栩栩如生——在弱条件下利用音频到视频扩散模型生成富有表现力的肖像视频
EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions
AI总结：
本文提出EMO框架，通过直接音频到视频的扩散合成方法，无需3D模型或面部标志点，即可生成高表现力、身份一致的说话和唱歌肖像视频，显著超越现有方法。
摘要：
在这项工作中，我们通过关注音频线索与面部动作之间动态而细微的关系，应对说话头部视频生成中提升真实感和表现力的挑战。我们指出了传统技术的局限性，这些技术往往无法捕捉人类表情的全部范围以及个体面部风格的独特性。为了解决这些问题，我们提出了EMO，一种新颖的框架，它利用直接的音频到视频合成方法，绕过了对中间3D模型或面部标志点的需求。我们的方法确保了视频中无缝的帧过渡和一致的身份保持，从而产生高度富有表现力和逼真的动画。实验结果表明，EMO不仅能够生成令人信服的说话视频，还能生成各种风格的唱歌视频，在表现力和真实感方面显著优于现有的最先进方法。
英文摘要：
In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
