用于社交远程呈现的音频驱动实时面部动画
Audio Driven Real-Time Facial Animation for Social Telepresence
AI总结：
提出一种音频驱动的实时面部动画系统，利用扩散模型与在线transformer及蒸馏管道实现低延迟、高精度的3D虚拟人动画，推理速度较离线基线快100至1000倍。
摘要：
我们提出了一种音频驱动的实时系统，以最小延迟为任何人绘制逼真的3D面部虚拟人动画，专为虚拟现实中的社交互动设计。我们方法的核心是一个编码器模型，实时将音频信号转换为潜在面部表情序列，随后解码为逼真的3D面部虚拟人。利用扩散模型的生成能力，我们捕捉自然交流所需的丰富面部表情，同时实现实时性能（<15ms GPU时间）。我们的新架构通过两项关键创新最小化延迟：一个消除对未来输入依赖的在线transformer，以及一个将迭代去噪加速为单步的蒸馏管道。我们进一步解决实时场景中逐帧处理连续音频信号同时保持一致动画质量的关键设计挑战。我们框架的多功能性扩展到多模态应用，包括语义模态（如情感条件）和VR头显上头戴式眼动摄像头的多模态传感器。实验结果表明，与现有离线最先进基线相比，面部动画精度显著提升，推理速度快100至1000倍。我们通过现场VR演示和多语言演讲等多种场景验证了该方法。
英文摘要：
We present an audio-driven real-time system for animating photorealistic 3D facial avatars with minimal latency, designed for social interactions in virtual reality for anyone. Central to our approach is an encoder model that transforms audio signals into latent facial expression sequences in real time, which are then decoded as photorealistic 3D facial avatars. Leveraging the generative capabilities of diffusion models, we capture the rich spectrum of facial expressions necessary for natural communication while achieving real-time performance (<15ms GPU time). Our novel architecture minimizes latency through two key innovations: an online transformer that eliminates dependency on future inputs and a distillation pipeline that accelerates iterative denoising into a single step. We further address critical design challenges in live scenarios for processing continuous audio signals frame-by-frame while maintaining consistent animation quality. The versatility of our framework extends to multimodal applications, including semantic modalities such as emotion conditions and multimodal sensors with head-mounted eye cameras on VR headsets. Experimental results demonstrate significant improvements in facial animation accuracy over existing offline state-of-the-art baselines, achieving 100 to 1000 times faster inference speed. We validate our approach through live VR demonstrations and across various scenarios such as multilingual speeches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
