基于情感和语义的对话手势合成系统
A conversational gesture synthesis system based on emotions and semantics
AI总结：
本文提出DeepGesture，一种基于扩散的 gesture 合成系统，通过多模态信号生成具有情感和语义条件的手势，提升数字人类的自然表达能力。
摘要：
随着大语言模型的爆炸性发展，语音合成的改进、硬件的进步以及计算机图形学的演进，当前创建数字人类的主要瓶颈在于生成与文本或语音输入自然对应的字符动作。在本工作中，我们提出了DeepGesture，一种基于扩散的 gesture 合成框架，用于根据多模态信号——文本、语音、情绪和种子动作——生成具有表现力的同步手势。基于DiffuseStyleGesture模型，DeepGesture引入了新的架构增强，以提高生成手势的语义对齐性和情感表现力。具体而言，我们集成了快速文本转录作为语义条件，并实现了情绪引导的分类器自由扩散，以支持在情感状态之间可控的手势生成。为了可视化结果，我们基于模型输出的BVH文件在Unity中实现了完整的渲染管线。在ZeroEGGS数据集上的评估显示，DeepGesture生成的手势在人类化和上下文适当性方面有所提高。我们的系统支持在情感状态之间进行插值，并展示了对分布外语音的泛化能力，包括合成语音——标志着向完全多模态、情感感知的数字人类迈进的一步。
英文摘要：
Along with the explosion of large language models, improvements in speech synthesis, advancements in hardware, and the evolution of computer graphics, the current bottleneck in creating digital humans lies in generating character movements that correspond naturally to text or speech inputs. In this work, we present DeepGesture, a diffusion-based gesture synthesis framework for generating expressive co-speech gestures conditioned on multimodal signals - text, speech, emotion, and seed motion. Built upon the DiffuseStyleGesture model, DeepGesture introduces novel architectural enhancements that improve semantic alignment and emotional expressiveness in generated gestures. Specifically, we integrate fast text transcriptions as semantic conditioning and implement emotion-guided classifier-free diffusion to support controllable gesture generation across affective states. To visualize results, we implement a full rendering pipeline in Unity based on BVH output from the model. Evaluation on the ZeroEGGS dataset shows that DeepGesture produces gestures with improved human-likeness and contextual appropriateness. Our system supports interpolation between emotional states and demonstrates generalization to out-of-distribution speech, including synthetic voices - marking a step forward toward fully multimodal, emotionally aware digital humans.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
