HM-Talker：用于高保真说话头合成的混合运动建模
HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis
AI总结：
提出HM-Talker框架，通过混合显式发音线索与隐式韵律特征，结合交叉模态映射和随机特征配对策略，解决说话头生成中个性化与泛化的权衡问题，在视觉真实感和唇同步精度上超越现有方法。
摘要：
音频驱动的说话头生成面临个性化与泛化之间的基本权衡，限制了其实际应用。隐式模型通常以结构不一致为代价实现泛化，导致不稳定的头部运动和不准确的唇同步。而显式方法引入了几何和解剖先验，如参数化面部几何的3D可变形模型（3DMM）或编码面部肌肉运动的动作单元（AU），但它们往往产生过度中性的表情或泛化能力有限。为解决这一矛盾，我们提出了HM-Talker，一个音频驱动的说话头框架，它协同整合显式发音线索与隐式韵律特征，以刻画身份特定动态，同时实现音频驱动的泛化。其显著特点可概括为：i) 跨模态映射模块（CMMM），从音频和视频中提取全面的运动线索词汇表；ii) 混合运动建模模块（HMMM），采用随机特征配对（SFP）策略，动态融合配对的隐式和显式特征以进行运动合成。该设计促进了下半部分面部运动的迭代优化，在身份特定目标与身份无关（仅音频）目标之间交替进行。大量实验表明，HM-Talker在多种设置下的视觉真实感和唇同步精度方面均优于最先进方法。
英文摘要：
Audio-driven talking head generation faces a fundamental trade-off between personalization and generalization, limiting its practical application. Implicit models often achieve generalization at the cost of structural incoherence, resulting in unstable head motion and inaccurate lip synchronization. While explicit methods incorporate geometric and anatomical priors such as 3D Morphable Models (3DMMs), which parameterize facial geometry, or Action Units (AUs), which code facial muscle movements--they tend to produce overly neutral expressions or suffer from limited generalization. To resolve this conflict, we present HM-Talker, an audio-driven talking head framework that synergistically integrates explicit articulatory cues with implicit prosodic features to characterize identity-specific dynamics while enabling audio-driven generalization. Its distinctive features can be summarized as: i) the Cross-Modal Mapping Module (CMMM) that extracts a comprehensive vocabulary of motion cues from audio and video, and ii) the Hybrid Motion Modeling Module (HMMM) that employs a Stochastic Feature Pairing (SFP) strategy to dynamically merge paired implicit and explicit features for motion synthesis. This design facilitates an iterative optimization of the lower face motion, alternating between identity-specific and identity-agnostic (audio-only) objectives. Extensive experiments demonstrate that HM-Talker outperforms state-of-the-art methods in both visual realism and lip-sync accuracy across diverse settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
