Gesticulator：一个语义感知的语音驱动手势生成框架
Gesticulator: A framework for semantically-aware speech-driven gesture generation
摘要：
人们在说话时会自发地做手势，手势在信息传递中起着关键作用。类似地，逼真的协同语音手势对于实现与社交代理的自然和流畅的交互至关重要。当前的端到端协同语音手势生成系统使用单个模态来表示语音：音频或文本。因此，这些系统被限制为产生声学链接的节拍手势或语义链接的手势（例如，在说“高”时举手）：他们不能适当地学习生成两种手势类型。我们提出了一个模型，旨在产生任意节拍和语义手势在一起。我们基于深度学习的模型将语音的声学和语义表示作为输入，并将手势生成为关节角度旋转序列作为输出。由此产生的手势可以应用于虚拟代理和人形机器人。主观和客观的评价证实了我们的方法的成功。代码和视频可以在项目页面https://svito-zar.github.io/gesticulator上找到。
英文摘要：
During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current end-to-end co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying "high"): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code and video are available at the project page https://svito-zar.github.io/gesticulator .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
