EchoMimic：通过可编辑 landmark 条件实现逼真的音频驱动肖像动画
EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
AI总结：
针对音频或单关键点驱动肖像动画存在的不稳定或不自然问题，提出可同时用音频、面部关键点及二者组合驱动的EchoMimic方法，经多数据集评估表现更优。
摘要：
在音频输入的推动下，肖像图像动画领域在生成逼真动态肖像方面取得了显著进展。传统方法仅限于利用音频或面部关键点来驱动图像生成视频，尽管它们能取得令人满意的结果，但仍存在一些问题。例如，仅由音频驱动的方法有时会因为音频信号相对较弱而不稳定；而仅由面部关键点驱动的方法，尽管驱动过程更稳定，但会因为关键点信息的过度控制导致生成结果不自然。为解决上述挑战，本文提出了一种名为EchoMimic的新方法。EchoMimic同时使用音频和面部landmark（关键点）进行训练。通过采用一种新颖的训练策略，EchoMimic不仅能够分别通过音频和面部landmark生成肖像视频，还能通过结合音频与选定的面部landmark来生成视频。我们在多个公开数据集以及我们收集的数据集上，将EchoMimic与其他算法进行了全面对比，结果表明其在定量和定性评估中均表现出更优的性能。更多可视化效果和源代码可在EchoMimic项目页面查看。
英文摘要：
The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
