GoHD：具备节律性姿态与逼真表情的面向注视且高度解耦的人像动画生成
GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
AI总结：
针对音频驱动说话人生成的挑战，提出GoHD框架，通过潜空间导航动画模块、conformer条件扩散模型和两阶段训练策略，实现高解耦、表情逼真且泛化性强的可控人像动画生成。
摘要：
音频驱动的说话人生成需要在多样化输入人像、以及音频与面部动作间复杂关联带来的挑战下，实现音频与视觉数据的无缝融合。针对这一问题，我们提出了鲁棒框架GoHD，旨在基于任意参考身份与任意动作，生成高度逼真、富有表现力且可控的人像视频。GoHD的创新点包含三个核心模块：首先，提出了采用潜空间导航的动画模块，以提升在未见过的输入风格上的泛化能力。该模块实现了动作与身份的高度解耦，还引入了注视方向以修正此前被忽视的不自然眼部运动。其次，设计了 conformer 结构的条件扩散模型，以确保头部姿态具备韵律感知能力。第三，为在有限训练数据下从输入音频估计出口型同步且逼真的表情，我们设计了两阶段训练策略，将频繁的逐帧唇部动作蒸馏，与其他更具时间依赖性但和音频关联较弱的动作（如眨眼、皱眉）的生成过程解耦。大量实验验证了GoHD出色的泛化能力，证明其在任意主体上生成逼真话人脸结果的有效性。
英文摘要：
Audio-driven talking head generation necessitates seamless integration of audio and visual data amidst the challenges posed by diverse input portraits and intricate correlations between audio and facial motions. In response, we propose a robust framework GoHD designed to produce highly realistic, expressive, and controllable portrait videos from any reference identity with any motion. GoHD innovates with three key modules: Firstly, an animation module utilizing latent navigation is introduced to improve the generalization ability across unseen input styles. This module achieves high disentanglement of motion and identity, and it also incorporates gaze orientation to rectify unnatural eye movements that were previously overlooked. Secondly, a conformer-structured conditional diffusion model is designed to guarantee head poses that are aware of prosody. Thirdly, to estimate lip-synchronized and realistic expressions from the input audio within limited training data, a two-stage training strategy is devised to decouple frequent and frame-wise lip motion distillation from the generation of other more temporally dependent but less audio-related motions, e.g., blinks and frowns. Extensive experiments validate GoHD's advanced generalization capabilities, demonstrating its effectiveness in generating realistic talking face results on arbitrary subjects.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
