VipDiff：基于免训练去噪扩散模型的连贯且多样视频修复
VipDiff: Towards Coherent and Diverse Video Inpainting via Training-free Denoising Diffusion Models
AI总结：
VipDiff提出免训练扩散模型框架，利用光流引导像素传播，实现时间连贯且多样的视频修复，显著优于现有方法。
摘要：
最近的视频修复方法通过利用光流引导像素从参考帧在图像空间或特征空间进行传播，取得了令人鼓舞的改进。然而，当掩蔽区域过大且无法为掩蔽中心找到像素对应关系时，这些方法会在掩蔽中心产生严重伪影。近年来，扩散模型在生成多样且高质量的图像方面展现了令人印象深刻的表现，并已被许多工作用于图像修复。然而，这些方法不能直接应用于视频以产生时间连贯的修复结果。在本文中，我们提出了一个名为VipDiff的免训练框架，用于在反向扩散过程中对扩散模型进行条件约束，以产生时间连贯的修复结果，而无需任何训练数据或微调预训练的扩散模型。VipDiff以光流作为指导，从参考帧提取有效像素作为约束，在优化随机采样的高斯噪声时使用，并利用生成的结果进行进一步的像素传播和条件生成。VipDiff还允许在不同采样的噪声上生成多样的视频修复结果。实验表明，VipDiff在空间-时间连贯性和保真度方面均大幅优于最先进的视频修复方法。
英文摘要：
Recent video inpainting methods have achieved encouraging improvements by leveraging optical flow to guide pixel propagation from reference frames either in the image space or feature space. However, they would produce severe artifacts in the mask center when the masked area is too large and no pixel correspondences can be found for the center. Recently, diffusion models have demonstrated impressive performance in generating diverse and high-quality images, and have been exploited in a number of works for image inpainting. These methods, however, cannot be applied directly to videos to produce temporal-coherent inpainting results. In this paper, we propose a training-free framework, named VipDiff, for conditioning diffusion model on the reverse diffusion process to produce temporal-coherent inpainting results without requiring any training data or fine-tuning the pre-trained diffusion models. VipDiff takes optical flow as guidance to extract valid pixels from reference frames to serve as constraints in optimizing the randomly sampled Gaussian noise, and uses the generated results for further pixel propagation and conditional generation. VipDiff also allows for generating diverse video inpainting results over different sampled noise. Experiments demonstrate that VipDiff can largely outperform state-of-the-art video inpainting methods in terms of both spatial-temporal coherence and fidelity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
