基于解耦潜在扩散的情感语音驱动3D人体动画
Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion
AI总结：
本文提出AMUSE模型，通过解耦潜在向量实现情感语音驱动的3D人体动画生成，能控制情绪和风格，生成更符合语音内容且表现更真实的情绪动作序列。
摘要：
现有方法在合成语音驱动的3D人体动作时未能显式建模情绪对动作的影响。为解决此问题，我们提出AMUSE模型，基于潜在扩散模型，将驱动音频映射为内容、情绪和风格三个解耦的潜在向量。训练后的模型能从语音直接生成具有可控情绪和风格的3D动作序列。随机采样扩散模型噪声进一步生成具有相同情绪表现的动画变体。定性、定量和感知评估表明，AMUSE生成的动画更真实，与语音内容同步更佳，情绪表达更准确。代码可在amuse.is.tue.mpg.de获取。
英文摘要：
Existing methods for synthesizing 3D human gestures from speech have shown promising results, but they do not explicitly model the impact of emotions on the generated gestures. Instead, these methods directly output animations from speech without control over the expressed emotion. To address this limitation, we present AMUSE, an emotional speech-driven body animation model based on latent diffusion. Our observation is that content (i.e., gestures related to speech rhythm and word utterances), emotion, and personal style are separable. To account for this, AMUSE maps the driving audio to three disentangled latent vectors: one for content, one for emotion, and one for personal style. A latent diffusion model, trained to generate gesture motion sequences, is then conditioned on these latent vectors. Once trained, AMUSE synthesizes 3D human gestures directly from speech with control over the expressed emotions and style by combining the content from the driving speech with the emotion and style of another speech sequence. Randomly sampling the noise of the diffusion model further generates variations of the gesture with the same emotional expressivity. Qualitative, quantitative, and perceptual evaluations demonstrate that AMUSE outputs realistic gesture sequences. Compared to the state of the art, the generated gestures are better synchronized with the speech content, and better represent the emotion expressed by the input speech. Our code is available at amuse.is.tue.mpg.de.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
