arXiv 2410.077572024-10-10cs.CV

MMHead：迈向细粒度多模态3D面部动画

MMHead: Towards Fine-grained Multi-modal 3D Facial Animation

Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）

Sijing Wu, Yunhao Li, Yichao Yan, Huiyu Duan, Ziwei Liu, Guangtao Zhai

备注：: Accepted by ACMMM 2024. Project page: https://wsj-sjtu.github.io/MMHead/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

为填补多模态3D面部动画数据集的空白，本文构建了大规模数据集MMHead并建立两项新基准，同时提出基于VQ-VAE的MM2Face方法统一多模态信息生成高质量3D面部运动。

摘要：

3D面部动画因其在多媒体领域的广泛应用而备受关注。音频驱动的3D面部动画已被广泛探索并取得了可喜的成果。然而，由于缺乏多模态3D面部动画数据集，多模态3D面部动画，尤其是文本引导的3D面部动画，鲜少被探索。为了填补这一空白，我们首先构建了一个大规模多模态3D面部动画数据集MMHead，它包含49小时的3D面部运动序列、语音音频和丰富的分层文本标注。每条文本标注包含抽象的动作与情感描述、细粒度的面部和头部运动（即表情和头部姿态）描述，以及三种可能引发此类情感的情景。具体而言，我们整合了五个公开的2D肖像视频数据集，并提出了一条自动化流程以：1）从单目视频中重建3D面部运动序列；以及2）借助AU检测和ChatGPT获取分层文本标注。基于MMHead数据集，我们为两项新任务建立了基准：文本诱导的3D说话头部动画和文本到3D面部运动生成。此外，提出了一种简单但高效的基于VQ-VAE的方法MM2Face，以统一多模态信息并生成多样且合理的3D面部运动，它在两个基准上均取得了具有竞争力的结果。大量实验和综合分析证明了我们的数据集和基准在推动多模态3D面部动画发展方面的巨大潜力。

英文摘要：

3D facial animation has attracted considerable attention due to its extensive applications in the multimedia field. Audio-driven 3D facial animation has been widely explored with promising results. However, multi-modal 3D facial animation, especially text-guided 3D facial animation is rarely explored due to the lack of multi-modal 3D facial animation dataset. To fill this gap, we first construct a large-scale multi-modal 3D facial animation dataset, MMHead, which consists of 49 hours of 3D facial motion sequences, speech audios, and rich hierarchical text annotations. Each text annotation contains abstract action and emotion descriptions, fine-grained facial and head movements (i.e., expression and head pose) descriptions, and three possible scenarios that may cause such emotion. Concretely, we integrate five public 2D portrait video datasets, and propose an automatic pipeline to 1) reconstruct 3D facial motion sequences from monocular videos; and 2) obtain hierarchical text annotations with the help of AU detection and ChatGPT. Based on the MMHead dataset, we establish benchmarks for two new tasks: text-induced 3D talking head animation and text-to-3D facial motion generation. Moreover, a simple but efficient VQ-VAE-based method named MM2Face is proposed to unify the multi-modal information and generate diverse and plausible 3D facial motions, which achieves competitive results on both benchmarks. Extensive experiments and comprehensive analysis demonstrate the significant potential of our dataset and benchmarks in promoting the development of multi-modal 3D facial animation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。