arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02827cs.CV
收藏

面向高保真图像到视频生成的免调优噪声校正方法

Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

Alimama Tech, Alibaba Group(阿里巴巴集团阿里妈妈技术)

Weijie Li, Litong Gong, Yiran Zhu, Fanda Fan, Biao Wang, Tiezheng Ge, Bo Zheng

AI总结:

针对开放域图像到视频生成保真度低的问题,提出一种免调优即插即用的噪声校正方法,通过给图像隐空间加噪保细节、去噪时校正偏差,有效提升生成视频的保真度,可适配主流视频扩散模型。

摘要:

图像到视频(I2V)生成任务在开放域场景下始终面临难以维持高保真度的问题。传统的图像动画技术主要聚焦于人脸、人体姿态等特定领域,难以泛化至开放域。近期部分基于扩散模型的I2V框架虽能为开放域图像生成动态内容,但无法维持保真度。我们研究发现,保真度较低的两个主要因素是图像细节丢失以及去噪过程中的噪声预测偏差。为此,我们提出一种可应用于主流视频扩散模型的有效方法,该方法通过补充更精准的图像信息与噪声校正实现高保真生成。具体而言,给定指定图像,我们的方法首先对输入图像的隐空间表示加噪以保留更多细节,随后对含噪隐表示进行去噪,并辅以适当校正以缓解噪声预测偏差。该方法无需调优,支持即插即用。实验结果证明了我们的方法在提升生成视频保真度方面的有效性。更多图像到视频的生成结果可参阅项目网站:https://noise-rectification.github.io。

英文摘要:

Image-to-video (I2V) generation tasks always suffer from keeping high fidelity in the open domains. Traditional image animation techniques primarily focus on specific domains such as faces or human poses, making them difficult to generalize to open domains. Several recent I2V frameworks based on diffusion models can generate dynamic content for open domain images but fail to maintain fidelity. We found that two main factors of low fidelity are the loss of image details and the noise prediction biases during the denoising process. To this end, we propose an effective method that can be applied to mainstream video diffusion models. This method achieves high fidelity based on supplementing more precise image information and noise rectification. Specifically, given a specified image, our method first adds noise to the input image latent to keep more details, then denoises the noisy latent with proper rectification to alleviate the noise prediction biases. Our method is tuning-free and plug-and-play. The experimental results demonstrate the effectiveness of our approach in improving the fidelity of generated videos. For more image-to-video generated results, please refer to the project website: https://noise-rectification.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑