MVPortrait：面向多视角生动肖像动画的文本引导运动与情感控制
MVPortrait: Text-Guided Motion and Emotion Control for Multi-view Vivid Portrait Animation
AI总结：
MVPortrait提出两阶段文本引导框架，首次以FLAME为中间表示，实现多视角肖像动画的运动与情感控制，并兼容文本、语音和视频驱动。
摘要：
近年来的肖像动画方法在生成逼真的唇形同步方面取得了显著进展。然而，这些方法通常缺乏对头部运动和面部表情的显式控制，并且无法从多个视角生成视频，导致动画的可控性和表现力不足。此外，尽管文本引导的肖像动画具有用户友好的特性，但其研究仍不充分。我们提出了一种新颖的两阶段文本引导框架MVPortrait（多视角生动肖像），用于生成能够忠实捕捉所描述运动和情感的表现力丰富的多视角肖像动画。MVPortrait首次引入FLAME作为中间表示，在其参数空间中有效嵌入面部运动、表情和视角变换。在第一阶段，我们分别基于文本输入训练FLAME运动扩散模型和情感扩散模型。在第二阶段，我们训练一个多视角视频生成模型，该模型以参考肖像图像和来自第一阶段的多视角FLAME渲染序列为条件。实验结果表明，MVPortrait在运动与情感控制以及视角一致性方面优于现有方法。此外，通过利用FLAME作为桥梁，MVPortrait成为首个兼容文本、语音和视频作为驱动信号的可控肖像动画框架。
英文摘要：
Recent portrait animation methods have made significant strides in generating realistic lip synchronization. However, they often lack explicit control over head movements and facial expressions, and cannot produce videos from multiple viewpoints, resulting in less controllable and expressive animations. Moreover, text-guided portrait animation remains underexplored, despite its user-friendly nature. We present a novel two-stage text-guided framework, MVPortrait (Multi-view Vivid Portrait), to generate expressive multi-view portrait animations that faithfully capture the described motion and emotion. MVPortrait is the first to introduce FLAME as an intermediate representation, effectively embedding facial movements, expressions, and view transformations within its parameter space. In the first stage, we separately train the FLAME motion and emotion diffusion models based on text input. In the second stage, we train a multi-view video generation model conditioned on a reference portrait image and multi-view FLAME rendering sequences from the first stage. Experimental results exhibit that MVPortrait outperforms existing methods in terms of motion and emotion control, as well as view consistency. Furthermore, by leveraging FLAME as a bridge, MVPortrait becomes the first controllable portrait animation framework that is compatible with text, speech, and video as driving signals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
