Freetalker：基于扩散模型的可控语音与文本驱动手势生成，以增强说话者自然度
Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness
AI总结：
针对现有虚拟人未考虑非说话动作及数据受限问题，提出FreeTalker框架，基于扩散模型统一语音与文本驱动动作，利用无分类器引导与DoubleTake方法生成自然可控且过渡平滑的说话者动作。
摘要：
当前的说话虚拟人大多基于话语的音频和文本生成协同语音手势，而未考虑说话者的非说话动作。此外，以往的协同语音手势生成工作基于单个手势数据集设计网络结构，这导致数据量有限、泛化性受损以及说话者动作受限。为了解决这些问题，我们提出了FreeTalker，据我们所知，这是首个同时生成自发（如协同语音手势）和非自发（如在讲台周围走动）说话者动作的框架。具体而言，我们训练了一个基于扩散模型的说话者动作生成模型，该模型采用语音驱动手势和文本驱动动作的统一表示，利用来自各种动作数据集的异构数据。在推理阶段，我们利用无分类器引导来高度控制片段中的风格。此外，为了在片段之间创建平滑过渡，我们利用DoubleTake方法，该方法利用生成先验并确保无缝的动作融合。大量实验表明，我们的方法能生成自然且可控的说话者动作。我们的代码、模型和演示可在\url{https://youngseng.github.io/FreeTalker/}获取。
英文摘要：
Current talking avatars mostly generate co-speech gestures based on audio and text of the utterance, without considering the non-speaking motion of the speaker. Furthermore, previous works on co-speech gesture generation have designed network structures based on individual gesture datasets, which results in limited data volume, compromised generalizability, and restricted speaker movements. To tackle these issues, we introduce FreeTalker, which, to the best of our knowledge, is the first framework for the generation of both spontaneous (e.g., co-speech gesture) and non-spontaneous (e.g., moving around the podium) speaker motions. Specifically, we train a diffusion-based model for speaker motion generation that employs unified representations of both speech-driven gestures and text-driven motions, utilizing heterogeneous data sourced from various motion datasets. During inference, we utilize classifier-free guidance to highly control the style in the clips. Additionally, to create smooth transitions between clips, we utilize DoubleTake, a method that leverages a generative prior and ensures seamless motion blending. Extensive experiments show that our method generates natural and controllable speaker movements. Our code, model, and demo are are available at \url{https://youngseng.github.io/FreeTalker/}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
