MoCoTalk: 多条件扩散与自适应路由的可控说话头生成
MoCoTalk: Multi-Conditional Diffusion with Adaptive Router for Controllable Talking Head Generation
AI总结：
MoCoTalk通过统一身份、头部姿态、面部表情和嘴部动态四个控制信号，提出自适应多条件路由框架，解决异构条件干扰问题，提升可控说话头生成的性能与可控性。
摘要：
说话头生成需要联合建模身份、头部姿态、面部表情和嘴部动态。现有方法通常只处理部分因素，且在多个条件时依赖固定权重或启发式融合。我们提出MoCoTalk，一种多条件视频扩散框架，统一四个互补的控制信号：参考图像、面部关键点、3DMM渲染的阴影网格和对应的语音音频。为解决异构条件间的破坏性干扰，我们引入自适应多条件路由，计算通道-wise、时间步-aware的门控，允许融合策略随特征子空间和噪声水平变化。为更好地捕捉语音相关的面部动态，我们设计了唇部增强的阴影网格，一种基于3DMM的表示，解耦头部运动、嘴部运动、表情和光照。此设计提供时间一致的几何先验，并允许在推理时灵活重组这些属性。我们进一步引入唇部一致性损失以收紧音视频对齐。大量实验表明，MoCoTalk在多数结构、运动和感知指标上达到最先进的性能，同时提供单条件方法无法提供的属性级可控性。
英文摘要：
Talking-head generation requires joint modeling of identity, head pose, facial expression, and mouth dynamics. Existing methods typically address only a subset of these factors, and rely on fixed-weight or heuristic fusion when multiple conditions are involved. We present MoCoTalk, a multi-conditional video diffusion framework that unifies four complementary control signals: a reference image, facial keypoints, 3DMM-rendered shading meshes, and the corresponding speech audio. To resolve destructive interference among heterogeneous conditions, we introduce an Adaptive Multi-Condition Router that computes channel-wise, timestep-aware gating over the four condition streams, allowing the fusion strategy to vary with both feature subspace and noise level. To better capture speech-related facial dynamics, we design a Mouth-Augmented Shading Mesh, a 3DMM-based representation that decouples head motion, mouth motion, expression, and lighting. This design provides a temporally consistent geometric prior and allows flexible recombination of these attributes at inference. We further introduce a lip consistency loss to tighten audio-visual alignment. Extensive experiments show that MoCoTalk achieves state-of-the-art performance on the majority of structural, motion, and perceptual metrics, while offering attribute-level controllability that single-condition methods do not provide.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
