识别并解决图像到视频扩散模型中的条件图像泄漏问题
Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
AI总结:
针对图像到视频扩散模型因条件图像泄漏导致运动不足的问题,提出从较早时间步生成并采用Analytic-Init初始噪声分布及TimeNoise训练噪声策略,在开放域基准和UCF101上提升运动分数与精度。
摘要:
扩散模型在图像到视频生成方面取得了显著进展。然而,在本文中,我们发现这些模型倾向于生成运动幅度低于预期的视频。我们将此归因于一个称为条件图像泄漏的问题,即图像到视频扩散模型(I2V-DMs)在大时间步长上倾向于过度依赖条件图像。我们进一步从推理和训练两个方面解决这一挑战。首先,我们提出从较早的时间步长开始生成过程,以避免I2V-DMs不可靠的大时间步长,同时通过最小化初始噪声分布与实际边际分布之间的KL散度,提出具有最优解析表达式(Analytic-Init)的初始噪声分布,以弥合训练与推理之间的差距。其次,我们在训练期间为条件图像设计了一种时间相关的噪声分布(TimeNoise),在较大的时间步长上施加更高的噪声水平以干扰条件图像,并减少模型对其的依赖。我们在收集的开放域图像基准和UCF101数据集上验证了这些通用策略在各种I2V-DMs上的有效性。大量结果表明,我们的方法通过产生更高的运动分数和更低的误差,同时保持图像对齐和时间一致性,优于基线方法,从而获得更优的整体性能并实现更精确的运动控制。项目页面:\url{https://cond-image-leak.github.io/}。
英文摘要:
Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。