TokTalk: 基于音频-大语言模型令牌的富有表现力的实时面部动画
TokTalk: Expressive Real-time Facial Animation from Audio-LLM Tokens
AI总结：
提出TokTalk系统，利用音频-大语言模型产生的音频令牌直接实时生成富有表现力的3D面部动画，通过分块条件流匹配模型和轻量级适配策略实现低延迟和高品质。
摘要：
近期GPT-4o等音频-大语言模型的进展开启了与语言模型对话交互的新时代。然而，对话式虚拟角色在面部表情和对话流程上仍显机械，部分原因在于其顺序执行语音识别、文本生成、轮次文本响应、语音合成和音频驱动面部动画等多个阶段。基于当前音频-大语言模型产生的音频令牌包含足够信息以重建合理面部表现这一洞察，我们提出TokTalk，一个直接从流式音频令牌实时输出富有表现力面部动画的系统。我们构建了一个新颖的音频令牌到3D面部运动数据集，并使用基于分块的条件流匹配模型训练TokTalk。一种轻量级适配策略使我们的训练模型能够以极小的计算开销无缝连接到任何基于令牌的音频-大语言模型。我们的分块处理进一步实现了延迟与面部质量之间的参数化权衡，并通过消融研究进行了验证。我们还表明，TokTalk的实时性能在延迟上与现有技术解决方案相当，而在3D面部表现的质量、表现力和可控性方面（通过感知研究）显著更优。我们通过聊天机器人虚拟角色、语音驱动的用户虚拟角色和动画导演界面展示了TokTalk在多种音视频面部应用中的灵活性。
英文摘要：
Recent advances in Audio-LLMs like GPT-4o have ushered in an era of conversational interaction with language models. Conversational avatars however, still seem robotic in facial expression and conversational flow, in part due to sequential stages of speech recognition, text generation, turn-based text response, speech synthesis, and audio driven facial animation. Based on our insight that audio-tokens produced by current Audio-LLMs carry sufficient information to reconstruct a plausible facial performance, we present TokTalk, a system that directly outputs expressive facial animation in real-time from streaming audio-tokens. We construct a novel audio-token to 3D facial motion dataset, on which TokTalk is trained using a Chunk-based Conditional Flow Matching model. A lightweight adaptation strategy allows our trained model to seamlessly connect to any token-based Audio-LLM at minimal computational overhead. Our chunk-based processing further enables parametric trade-off between latency and facial quality, shown through ablation studies. We further show that the real-time performance of TokTalk is comparable in latency to prior art solutions, and significantly favorable (via a perceptual study) in terms of quality, expressivity and control of the 3D facial performance. We showcase TokTalk's flexibility using a chatbot Avatar, a voice-driven user Avatar, and an animation Director's interface, as diverse audio-visual face applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
