EmoTalk3D：情感化3D说话头部的高保真自由视角合成
EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head
AI总结：
本文提出EmoTalk3D框架与配套多视角数据集，通过“语音-几何-外观”映射及4D高斯解耦，实现情感可控、高保真且自由视角的3D说话头部合成。
摘要：
我们提出了一种用于合成具有可控情感的3D说话头部的新颖方法，其特点是增强了唇形同步和渲染质量。尽管该领域已取得显著进展，但以往方法仍存在多视角一致性不足以及情感表达能力缺乏的问题。为解决这些问题，我们收集了EmoTalk3D数据集，其中包含经过标定的多视角视频、情感标注以及逐帧3D几何。通过在EmoTalk3D数据集上训练，我们提出了一个“语音到几何再到外观”的映射框架：该框架首先根据音频特征预测忠实的3D几何序列，随后由预测的几何合成以4D高斯表示的3D说话头部外观。外观进一步被解耦为规范高斯和动态高斯，二者从多视角视频中学习，并被融合以渲染自由视角的说话头部动画。此外，我们的模型能够控制所生成说话头部中的情感，并可在大视角范围内渲染。我们的方法在捕捉皱纹和细微表情等动态面部细节的同时，提升了渲染质量以及唇部运动生成的稳定性。实验证明了我们的方法在生成高保真且情感可控的3D说话头部方面的有效性。代码和EmoTalk3D数据集已发布于https://nju-3dv.github.io/projects/EmoTalk3D。
英文摘要：
We present a novel approach for synthesizing 3D talking heads with controllable emotion, featuring enhanced lip synchronization and rendering quality. Despite significant progress in the field, prior methods still suffer from multi-view consistency and a lack of emotional expressiveness. To address these issues, we collect EmoTalk3D dataset with calibrated multi-view videos, emotional annotations, and per-frame 3D geometry. By training on the EmoTalk3D dataset, we propose a \textit{`Speech-to-Geometry-to-Appearance'} mapping framework that first predicts faithful 3D geometry sequence from the audio features, then the appearance of a 3D talking head represented by 4D Gaussians is synthesized from the predicted geometry. The appearance is further disentangled into canonical and dynamic Gaussians, learned from multi-view videos, and fused to render free-view talking head animation. Moreover, our model enables controllable emotion in the generated talking heads and can be rendered in wide-range views. Our method exhibits improved rendering quality and stability in lip motion generation while capturing dynamic facial details such as wrinkles and subtle expressions. Experiments demonstrate the effectiveness of our approach in generating high-fidelity and emotion-controllable 3D talking heads. The code and EmoTalk3D dataset are released at https://nju-3dv.github.io/projects/EmoTalk3D.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
