RAP: 基于音频的实时人物动画与视频扩散变换器
RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer
AI总结：
RAP通过混合注意力机制和静态-动态训练-推理范式，在实时约束下实现高质量音频驱动人物动画，提升音频-视觉同步与视觉保真度。
摘要：
基于音频的人物动画旨在从输入音频信号和单张参考图像合成逼真且自然的说话人视频。尽管现有方法通过利用高维中间表示和显式建模运动动态实现了高质量的结果，但其计算复杂性使其不适合实时部署。实时推理对延迟和内存提出了严格限制，通常需要使用高度压缩的潜在表示。然而，在这种紧凑的空间中操作会阻碍细粒度时空细节的保留，从而复杂化音频-视觉同步。RAP（实时基于音频的人物动画），一个在实时约束下生成高质量说话人图像的统一框架。具体而言，RAP引入了一种混合注意力机制用于细粒度音频控制，并采用静态-动态训练-推理范式以避免显式运动监督。通过这些技术，RAP实现了精确的音频驱动控制，缓解了长期时间漂移，并保持了高质量的视觉保真度。大量实验表明，RAP在实时约束下实现了最先进的性能。
英文摘要：
Audio-driven portrait animation aims to synthesize realistic and natural talking head videos from an input audio signal and a single reference image. While existing methods achieve high-quality results by leveraging high-dimensional intermediate representations and explicitly modeling motion dynamics, their computational complexity renders them unsuitable for real-time deployment. Real-time inference imposes stringent latency and memory constraints, often necessitating the use of highly compressed latent representations. However, operating in such compact spaces hinders the preservation of fine-grained spatiotemporal details, thereby complicating audio-visual synchronization RAP (Real-time Audio-driven Portrait animation), a unified framework for generating high-quality talking portraits under real-time constraints. Specifically, RAP introduces a hybrid attention mechanism for fine-grained audio control, and a static-dynamic training-inference paradigm that avoids explicit motion supervision. Through these techniques, RAP achieves precise audio-driven control, mitigates long-term temporal drift, and maintains high visual fidelity. Extensive experiments demonstrate that RAP achieves state-of-the-art performance while operating under real-time constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
