M2DAO-Talker：协调多粒度运动解耦与交替优化的说话人脸生成
M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation
AI总结：
本文提出统一框架M2DAO-Talker，通过多粒度运动解耦与交替优化策略，独立建模刚性与非刚性运动并施加运动一致性约束，有效解决了说话人脸生成中的渲染伪影问题，在多数据集上实现SOTA且推理速度达150 FPS。
摘要：
音频驱动的说话人脸生成在电影制作中具有巨大潜力。尽管现有的3D方法在运动建模和内容合成方面取得了进展，但由于在表示稳定、细粒度运动场方面的局限性，它们常产生运动模糊、时间抖动和局部穿透等渲染伪影。通过系统分析，我们将说话人脸生成重新表述为一个包含三个步骤的统一框架：视频预处理、运动表示和渲染重建。该框架支撑了我们提出的M2DAO-Talker，它通过多粒度运动解耦和交替优化来解决当前的局限性。具体而言，我们设计了一种新颖的2D肖像预处理流水线，以提取逐帧形变控制条件（运动区域分割掩码和相机参数），从而促进运动表示。为了改善运动建模，我们详细阐述了一种多粒度运动解耦策略，该策略独立地对非刚性（口腔和面部）和刚性（头部）运动进行建模，以提高重建精度。同时，开发了运动一致性约束以确保头颈躯干运动学一致性，从而减轻由运动混叠引起的穿透伪影。此外，设计了一种交替优化策略，以迭代地细化面部和口腔运动参数，实现更逼真的视频生成。跨多个数据集的实验表明，M2DAO-Talker实现了最先进的性能，与TalkingGaussian相比，生成质量提高了2.43 dB PSNR，用户评估的视频真实度提升了0.64，同时推理速度达到150 FPS。我们的项目主页是https://m2dao-talker.github.io/M2DAO-Talk.github.io。
英文摘要：
Audio-driven talking head generation holds significant potential for film production. While existing 3D methods have advanced motion modeling and content synthesis, they often produce rendering artifacts, such as motion blur, temporal jitter, and local penetration, due to limitations in representing stable, fine-grained motion fields. Through systematic analysis, we reformulate talking head generation into a unified framework comprising three steps: video preprocessing, motion representation, and rendering reconstruction. This framework underpins our proposed M2DAO-Talker, which addresses current limitations via multi-granular motion decoupling and alternating optimization. Specifically, we devise a novel 2D portrait preprocessing pipeline to extract frame-wise deformation control conditions (motion region segmentation masks, and camera parameters) to facilitate motion representation. To ameliorate motion modeling, we elaborate a multi-granular motion decoupling strategy, which independently models non-rigid (oral and facial) and rigid (head) motions for improved reconstruction accuracy. Meanwhile, a motion consistency constraint is developed to ensure head-torso kinematic consistency, thereby mitigating penetration artifacts caused by motion aliasing. In addition, an alternating optimization strategy is designed to iteratively refine facial and oral motion parameters, enabling more realistic video generation. Experiments across multiple datasets show that M2DAO-Talker achieves state-of-the-art performance, with the 2.43 dB PSNR improvement in generation quality and 0.64 gain in user-evaluated video realness versus TalkingGaussian while with 150 FPS inference speed. Our project homepage is https://m2dao-talker.github.io/M2DAO-Talk.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
