面向实时语音同步视频生成的输入感知稀疏注意力
Input-Aware Sparse Attention for Real-Time Co-Speech Video Generation
AI总结：
针对现有扩散式语音同步视频生成方法速度慢难实时的问题，提出结合输入人体姿态的稀疏注意力与蒸馏损失，实现实时生成质量更优的语音同步视频。
摘要：
扩散模型可从音频合成逼真的语音同步视频，应用于视频创作、虚拟智能体等场景。但现有基于扩散的方法因大量去噪步骤和高开销注意力机制导致推理速度慢，无法实现实时部署。本研究将多步扩散视频模型蒸馏为少步学生模型，但直接应用现有扩散蒸馏方法会降低视频质量且达不到实时性能。为解决这些问题，我们提出的新型视频蒸馏方法在注意力机制和损失函数中均利用输入人体姿态条件。首先提出利用输入人体姿态关键点间的精准对应关系，引导注意力聚焦相关区域（如说话者的面部、手部和上半身），这种输入感知稀疏注意力减少了冗余计算，增强了身体部位的时序对应性，提升了推理效率和动作连贯性。为进一步提升视觉质量，我们引入输入感知蒸馏损失，改善唇部同步效果和手部动作真实感。通过整合输入感知稀疏注意力和蒸馏损失，我们的方法实现了实时性能，且视觉质量优于近期的音频驱动和输入驱动方法。我们还开展了大量实验，证明了算法设计选择的有效性。
英文摘要：
Diffusion models can synthesize realistic co-speech video from audio for various applications, such as video creation and virtual agents. However, existing diffusion-based methods are slow due to numerous denoising steps and costly attention mechanisms, preventing real-time deployment. In this work, we distill a many-step diffusion video model into a few-step student model. Unfortunately, directly applying recent diffusion distillation methods degrades video quality and falls short of real-time performance. To address these issues, our new video distillation method leverages input human pose conditioning for both attention and loss functions. We first propose using accurate correspondence between input human pose keypoints to guide attention to relevant regions, such as the speaker's face, hands, and upper body. This input-aware sparse attention reduces redundant computations and strengthens temporal correspondences of body parts, improving inference efficiency and motion coherence. To further enhance visual quality, we introduce an input-aware distillation loss that improves lip synchronization and hand motion realism. By integrating our input-aware sparse attention and distillation loss, our method achieves real-time performance with improved visual quality compared to recent audio-driven and input-driven methods. We also conduct extensive experiments showing the effectiveness of our algorithmic design choices.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
