arXiv 2511.224882025-11-27cs.CV

AI杀死了视频明星。用于表达性说话头生成的音频驱动扩散模型

AI killed the video star. Audio-driven diffusion model for expressive talking head generation

Centre INRIA d’Université Côte d’Azur（INRIA中心，坎特伯雷大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

Baptiste Chopin, Tashvik Dhamija, Pranav Balaji, Yaohui Wang, Antitza Dantcheva

备注：: arXiv admin note: text overlap with arXiv:2502.17198

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

Dimitra++通过音频驱动的扩散模型生成具有真实嘴唇运动、面部表情和头部姿态的说话头，优于现有方法。

摘要：

我们提出Dimitra++，一种新的音频驱动说话头生成框架，旨在学习嘴唇运动、面部表情以及头部姿态运动。具体来说，我们提出了一种条件运动扩散变换器（cMDT）来建模面部运动序列，采用三维表示。cMDT的条件输入包括两个：一个参考面部图像，确定外观，以及一个音频序列，驱动运动。定量和定性实验，以及在两个广泛使用的数据集，即VoxCeleb2和CelebV-HQ上的用户研究，表明Dimitra++能够在生成真实说话头方面优于现有方法，这些说话头具有嘴唇运动、面部表情和头部姿态。

英文摘要：

We propose Dimitra++, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we propose a conditional Motion Diffusion Transformer (cMDT) to model facial motion sequences, employing a 3D representation. The cMDT is conditioned on two inputs: a reference facial image, which determines appearance, as well as an audio sequence, which drives the motion. Quantitative and qualitative experiments, as well as a user study on two widely employed datasets, i.e., VoxCeleb2 and CelebV-HQ, suggest that Dimitra++ is able to outperform existing approaches in generating realistic talking heads imparting lip motion, facial expression, and head pose.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。