MMTalker: 多分辨率3D说话头合成与多模态特征融合
MMTalker: Multiresolution 3D Talking Head Synthesis with Multimodal Feature Fusion
AI总结：
提出一种基于多分辨率表示和多模态特征融合的3D语音驱动面部动画合成方法MMTalker，通过网格参数化、非均匀可微采样、残差图卷积网络和双交叉注意力机制，实现高唇同步精度和逼真面部表情。
摘要：
语音驱动的三维（3D）面部动画合成旨在建立从一维（1D）语音信号到时变3D面部运动信号的映射。当前方法在保持唇同步精度和生成逼真面部表情方面仍面临挑战，主要由于这种跨模态映射的高度病态性。本文通过多分辨率表示和多模态特征融合，提出一种新颖的3D音频驱动面部动画合成方法MMTalker，能够准确重建3D面部运动的丰富细节。我们首先通过网格参数化和非均匀可微采样实现带有细节的3D面部连续表示。网格参数化技术建立了UV平面与3D面部网格之间的对应关系，并用于为连续学习提供真值。可微非均匀采样通过在每个三角面中设置可学习的采样概率，实现精确的面部细节获取。接着，我们采用残差图卷积网络和双交叉注意力机制，从多个输入模态中提取判别性面部运动特征。所提出的多模态融合策略充分利用了语音的分层特征和面部网格的显式时空几何特征。最后，一个轻量级回归网络通过联合处理规范UV空间中的采样点和编码的面部运动特征，预测合成说话头的逐顶点几何位移。综合实验表明，与现有最先进方法相比，该方法在唇部和眼部运动的同步精度上取得了显著提升。
英文摘要：
Speech-driven three-dimensional (3D) facial animation synthesis aims to build a mapping from one-dimensional (1D) speech signals to time-varying 3D facial motion signals. Current methods still face challenges in maintaining lip-sync accuracy and producing realistic facial expressions, primarily due to the highly ill-posed nature of this cross-modal mapping. In this paper, we introduce a novel 3D audio-driven facial animation synthesis method through multi-resolution representation and multi-modal feature fusion, called MMTalker which can accurately reconstruct the rich details of 3D facial motion. We first achieve the continuous representation of 3D face with details by mesh parameterization and non-uniform differentiable sampling. The mesh parameterization technique establishes the correspondence between UV plane and 3D facial mesh and is used to offer ground truth for the continuous learning. Differentiable non-uniform sampling enables precise facial detail acquisition by setting learnable sampling probability in each triangular face. Next, we employ residual graph convolutional network and dual cross-attention mechanism to extract discriminative facial motion feature from multiple input modalities. This proposed multimodal fusion strategy takes full use of the hierarchical features of speech and the explicit spatiotemporal geometric features of facial mesh. Finally, a lightweight regression network predicts the vertex-wise geometric displacements of the synthesized talking face by jointly processing the sampled points in the canonical UV space and the encoded facial motion features. Comprehensive experiments demonstrate that significant improvements are achieved over state-of-the-art methods, especially in the synchronization accuracy of lip and eye movements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
