Dimitra：用于表现力说话头生成的音频驱动扩散模型
Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
AI总结：
Dimitra是一个音频驱动的扩散模型框架，通过条件运动扩散Transformer和3D建模，利用音频和参考图像生成具有嘴唇运动、面部表情和头部姿态的逼真说话头，在VoxCeleb2和HDTF上超越现有方法。
摘要：
我们提出了Dimitra，一种用于音频驱动说话头生成的新框架，其设计旨在学习嘴唇运动、面部表情以及头部姿态运动。具体而言，我们通过使用3D表示对面部运动序列进行建模，训练了一个条件运动扩散Transformer（cMDT）。我们仅用两个输入信号来条件化cMDT，即一个音频序列和一张参考面部图像。通过直接从音频中提取额外特征，Dimitra能够提高生成视频的质量和真实感。特别是，音素序列有助于嘴唇运动的真实感，而文本转录则有助于面部表情和头部姿态的真实感。在两个广泛使用的数据集VoxCeleb2和HDTF上进行的定量和定性实验表明，Dimitra在生成具有嘴唇运动、面部表情和头部姿态的真实说话头方面能够超越现有方法。
英文摘要：
We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
