arXiv 2602.105162026-04-03cs.CV

3DXTalker：统一身份、唇部同步、情绪和空间动态的表达性3D对话虚拟角色

3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

University of New South Wales（新南威尔士大学）；Australian National University（澳大利亚国立大学）；Vertex Lab（Vertex实验室）；University of Technology Sydney（悉尼科技大学）

Zhongju Wang, Zhenhong Sun, Beier Wang, Yifu Wang, Daoyi Dong, Huadong Mo, Hongdong Li

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出3DXTalker，通过数据精选的身份建模、音频丰富表示和空间动态可控性，统一实现身份、唇部同步、情绪和空间动态，提升3D对话虚拟角色的表达性。

摘要：

音频驱动的3D对话虚拟角色生成在虚拟通信、数字人类和互动媒体中日益重要，其中虚拟角色必须保持身份、同步唇部运动与语音、表达情绪并展现逼真的空间动态，共同定义了表达性的更广目标。然而，由于训练数据不足、主体身份有限、音频表示狭窄和显式可控性受限，实现这一目标仍具挑战性。本文提出3DXTalker，一种通过数据精选的身份建模、音频丰富表示和空间动态可控性实现的表达性3D对话虚拟角色。3DXTalker通过2D到3D数据精选流程和解耦表示实现可扩展的身份建模，缓解数据稀缺并提高身份泛化能力。然后，我们引入帧级幅度和情绪线索，超越标准语音嵌入，确保卓越的唇部同步和细腻的表情调节。这些线索通过基于流匹配的变压器统一，实现连贯的面部动态。此外，3DXTalker还能生成自然的头部姿态运动，并通过基于提示的条件控制支持风格化控制。广泛实验表明，3DXTalker在统一框架中整合唇部同步、情绪表达和头部姿态动态，实现了3D对话虚拟角色生成的优越性能。

英文摘要：

Audio-driven 3D talking avatar generation is increasingly important in virtual communication, digital humans, and interactive media, where avatars must preserve identity, synchronize lip motion with speech, express emotion, and exhibit lifelike spatial dynamics, collectively defining a broader objective of expressivity. However, achieving this remains challenging due to insufficient training data with limited subject identities, narrow audio representations, and restricted explicit controllability. In this paper, we propose 3DXTalker, an expressive 3D talking avatar through data-curated identity modeling, audio-rich representations, and spatial dynamics controllability. 3DXTalker enables scalable identity modeling via 2D-to-3D data curation pipeline and disentangled representations, alleviating data scarcity and improving identity generalization. Then, we introduce frame-wise amplitude and emotional cues beyond standard speech embeddings, ensuring superior lip synchronization and nuanced expression modulation. These cues are unified by a flow-matching-based transformer for coherent facial dynamics. Moreover, 3DXTalker also enables natural head-pose motion generation while supporting stylized control via prompt-based conditioning. Extensive experiments show that 3DXTalker integrates lip synchronization, emotional expression, and head-pose dynamics within a unified framework, achieves superior performance in 3D talking avatar generation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。