Teller：基于自回归运动生成的实时流式音频驱动人像动画
Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation
AI总结：
本研究提出首个自回归流式音频驱动人像动画框架Teller，通过面部运动潜变量生成与高效时序优化模块，实现25 FPS实时性能，推理速度远超扩散模型，在小幅运动逼真度上显著优于现有方法。
摘要：
在本研究中，我们提出了首个用于实时音频驱动人像动画（即数字人说话头部生成）的自回归框架。除了动画生成耗时长的挑战外，逼真的说话头部生成的一个关键难点在于保留身体各部位的自然运动。为此，我们提出了Teller——首个具备自回归运动生成能力的流式音频驱动人像动画框架。具体而言，Teller首先将面部与身体细节动画分解为两个模块：基于自回归Transformer的面部运动潜变量生成模块（Facial Motion Latent Generation, FMLG），以及使用高效时序模块（Efficient Temporal Module, ETM）的运动真实性优化模块。具体来说，FMLG采用残差VQ模型，将基于隐式关键点的模型输出的面部运动潜变量映射为离散运动标记，再将这些标记与音频嵌入进行时序切片，使自回归Transformer能够学习从音频到运动的实时流式映射。此外，Teller融入ETM以捕捉更精细的运动细节，该模块确保颈部肌肉、耳环等身体部位与配饰的物理一致性，提升这些运动的逼真度。Teller设计高效，生成1秒视频的推理速度超过扩散模型（Hallo需20.93秒，Teller仅需0.92秒），实时流式性能最高可达25 FPS。大量实验表明，我们的方法优于近期的音频驱动人像动画模型，尤其在小幅运动表现上，人类评估结果验证其在质量与逼真度上均有显著优势。
英文摘要：
In this work, we introduce the first autoregressive framework for real-time, audio-driven portrait animation, a.k.a, talking head. Beyond the challenge of lengthy animation times, a critical challenge in realistic talking head generation lies in preserving the natural movement of diverse body parts. To this end, we propose Teller, the first streaming audio-driven protrait animation framework with autoregressive motion generation. Specifically, Teller first decomposes facial and body detail animation into two components: Facial Motion Latent Generation (FMLG) based on an autoregressive transfromer, and movement authenticity refinement using a Efficient Temporal Module (ETM).Concretely, FMLG employs a Residual VQ model to map the facial motion latent from the implicit keypoint-based model into discrete motion tokens, which are then temporally sliced with audio embeddings. This enables the AR tranformer to learn real-time, stream-based mappings from audio to motion. Furthermore, Teller incorporate ETM to capture finer motion details. This module ensures the physical consistency of body parts and accessories, such as neck muscles and earrings, improving the realism of these movements. Teller is designed to be efficient, surpassing the inference speed of diffusion-based models (Hallo 20.93s vs. Teller 0.92s for one second video generation), and achieves a real-time streaming performance of up to 25 FPS. Extensive experiments demonstrate that our method outperforms recent audio-driven portrait animation models, especially in small movements, as validated by human evaluations with a significant margin in quality and realism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
