Soundini：面向自然视频编辑的声音引导扩散方法
Soundini: Sound-Guided Diffusion for Natural Video Editing
AI总结：
本文提出Soundini方法，首次在零样本设定下利用声音潜在表示引导扩散模型进行自然视频编辑，并结合光流引导保证帧间时间一致性，从而生成反映声音强度、音色和音量等属性的动态视觉特效。
摘要：
我们提出了一种在零样本设定下为视频特定区域添加声音引导视觉特效的方法。为视觉特效的外观制作动画具有挑战性，因为编辑后视频的每一帧都应在保持时间一致性的同时产生视觉变化。此外，现有视频编辑方案侧重于跨帧的时间一致性，忽略了随时间变化的视觉风格变化，例如雷暴、波浪、火焰噼啪声。为克服这一局限，我们利用时间声音特征来实现动态风格。具体而言，我们在音视频潜在空间中使用音频潜在表示来引导去噪扩散概率模型。据我们所知，我们的工作是首次探索从各种声源出发、利用声音特有属性（如强度、音色和音量）进行声音引导的自然视频编辑。此外，我们设计了基于光流的引导来生成时间一致的视频帧，捕捉相邻帧之间的像素级关系。实验结果表明，我们的方法优于现有视频编辑技术，能够生成更逼真且反映声音属性的视觉特效。请访问我们的页面：https://kuai-lab.github.io/soundini-gallery/。
英文摘要：
We propose a method for adding sound-guided visual effects to specific regions of videos with a zero-shot setting. Animating the appearance of the visual effect is challenging because each frame of the edited video should have visual changes while maintaining temporal consistency. Moreover, existing video editing solutions focus on temporal consistency across frames, ignoring the visual style variations over time, e.g., thunderstorm, wave, fire crackling. To overcome this limitation, we utilize temporal sound features for the dynamic style. Specifically, we guide denoising diffusion probabilistic models with an audio latent representation in the audio-visual latent space. To the best of our knowledge, our work is the first to explore sound-guided natural video editing from various sound sources with sound-specialized properties, such as intensity, timbre, and volume. Additionally, we design optical flow-based guidance to generate temporally consistent video frames, capturing the pixel-wise relationship between adjacent frames. Experimental results show that our method outperforms existing video editing techniques, producing more realistic visual effects that reflect the properties of sound. Please visit our page: https://kuai-lab.github.io/soundini-gallery/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
