arXiv 2410.172622025-05-01cs.CVcs.AIcs.HCcs.LG

EmoGene：音频驱动的情感3D说话人脸生成

EmoGene: Audio-Driven Emotional 3D Talking-Head Generation

Khoury College of Computer Science, Northeastern University, USA（东北大学Khoury计算机科学学院）；Department of Electrical and Computer Engineering, Northeastern University, USA（东北大学电气与计算机工程系）

Wenqing Wang, Yun Fu

备注：: Accepted by the 2025 IEEE 19th International Conference on Automatic Face and Gesture Recognition (FG)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出EmoGene框架，通过结合基于VAE的音频到运动模块和基于NeRF的情感到视频模块，生成高保真且具有准确情感表情的音频驱动3D说话人脸视频。

摘要：

音频驱动的说话人脸生成是虚拟人交互和电影制作中一项关键且有用的技术。尽管近期进展主要集中在提升图像保真度和唇部同步上，但生成准确的情感表情仍未被充分探索。在本文中，我们提出了EmoGene，一个用于合成高保真、音频驱动且带有准确情感表情的视频肖像的新颖框架。我们的方法采用基于变分自编码器（VAE）的音频到运动模块来生成面部关键点，这些关键点在运动到情感模块中与情感嵌入拼接以生成情感关键点。这些关键点驱动基于神经辐射场（NeRF）的情感到视频模块，渲染出逼真的情感说话人脸视频。此外，我们提出了一种姿态采样方法，为静音音频输入生成自然的空闲状态（非说话）视频。大量实验表明，EmoGene在生成高保真情感说话人脸视频方面优于先前的方法。

英文摘要：

Audio-driven talking-head generation is a crucial and useful technology for virtual human interaction and film-making. While recent advances have focused on improving image fidelity and lip synchronization, generating accurate emotional expressions remains underexplored. In this paper, we introduce EmoGene, a novel framework for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. Our approach employs a variational autoencoder (VAE)-based audio-to-motion module to generate facial landmarks, which are concatenated with emotional embedding in a motion-to-emotion module to produce emotional landmarks. These landmarks drive a Neural Radiance Fields (NeRF)-based emotion-to-video module to render realistic emotional talking-head videos. Additionally, we propose a pose sampling method to generate natural idle-state (non-speaking) videos for silent audio inputs. Extensive experiments demonstrate that EmoGene outperforms previous methods in generating high-fidelity emotional talking-head videos.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。