arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2504.09656cs.CV
收藏

KeyVID:用于音频同步视觉动画的关键帧感知视频扩散

KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animation

Xingrui Wang, Jiang Liu, Ze Wang, Xiaodong Yu, Jialian Wu, Ximeng Sun, Yusheng Su, Alan Yuille, Zicheng Liu, Emad Barsoum

AI总结:

提出关键帧感知的音频到视觉动画框架KeyVID,通过从音频定位关键帧并生成,再用运动插值器补全中间帧,显著提升高动态场景下的音视频同步与视频质量。

摘要:

从各种条件(如文本、图像和音频)生成视频,能够实现空间和时间上的控制,从而产生高质量的生成结果。具有剧烈运动的视频通常需要更高的帧率来确保运动平滑。目前,大多数音频到视觉动画模型使用从视频片段中均匀采样的帧。然而,这些均匀采样的帧在低帧率下无法捕捉剧烈运动中的关键瞬间,并且直接增加帧数会显著增加内存消耗。在本文中,我们提出了KeyVID,一个关键帧感知的音频到视觉动画框架,它显著提高了音频信号中关键时刻的生成质量,同时保持了计算效率。给定图像和音频输入,我们首先从音频中定位关键帧时间步。然后,我们使用关键帧生成器生成相应的视觉关键帧。最后,我们使用运动插值器生成所有中间帧。通过大量实验,我们证明KeyVID在多个数据集上显著提高了音视频同步和视频质量,特别是在高动态运动方面。代码已发布在https://github.com/XingruiWang/KeyVID。

英文摘要:

Generating video from various conditions, such as text, image, and audio, enables both spatial and temporal control, leading to high-quality generation results. Videos with dramatic motions often require a higher frame rate to ensure smooth motion. Currently, most audio-to-visual animation models use uniformly sampled frames from video clips. However, these uniformly sampled frames fail to capture significant key moments in dramatic motions at low frame rates and require significantly more memory when increasing the number of frames directly. In this paper, we propose KeyVID, a keyframe-aware audio-to-visual animation framework that significantly improves the generation quality for key moments in audio signals while maintaining computation efficiency. Given an image and an audio input, we first localize keyframe time steps from the audio. Then, we use a keyframe generator to generate the corresponding visual keyframes. Finally, we generate all intermediate frames using the motion interpolator. Through extensive experiments, we demonstrate that KeyVID significantly improves audio-video synchronization and video quality across multiple datasets, particularly for highly dynamic motions. The code is released in https://github.com/XingruiWang/KeyVID.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑