arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.09605cs.CV
收藏

DreamMover:利用扩散模型先验实现大运动图像插值

DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion

School of AIA, Huazhong University of Science and Technology(华中科技大学AIA学院);Adobe Research(奥多比研究院)

Liao Shen, Tianqi Liu, Huiqiang Sun, Xinyi Ye, Baopu Li, Jianming Zhang, Zhiguo Cao

AI总结:

DreamMover利用扩散模型的先验知识,通过自然流估计、分层融合和自注意力替换,解决大运动图像插值中的语义一致性问题,并引入InterpBench基准验证效果。

摘要:

我们研究从具有大运动的图像对生成中间图像并保持语义一致性的问题。由于运动幅度大,中间语义信息可能不存在于输入图像中。现有方法要么局限于小运动,要么专注于拓扑相似的对象,导致插值结果中出现伪影和不一致。为了克服这一挑战,我们深入研究了预训练图像扩散模型在语义认知和表示方面的能力,确保缺失的中间语义表示与输入保持一致表达。为此,我们提出了DreamMover,一种新颖的图像插值框架,包含三个主要组成部分:1)基于扩散模型的自然流估计器,能够隐式推理两幅图像之间的语义对应关系。2)为避免融合过程中细节信息的丢失,我们的关键见解是在高层空间和低层空间两部分中融合信息。3)为增强生成图像与输入之间的一致性,我们提出了自注意力拼接与替换方法。最后,我们提出了一个具有挑战性的基准数据集InterpBench,用于评估生成结果的语义一致性。大量实验证明了我们方法的有效性。我们的项目可在https://dreamm0ver.github.io获取。

英文摘要:

We study the problem of generating intermediate images from image pairs with large motion while maintaining semantic consistency. Due to the large motion, the intermediate semantic information may be absent in input images. Existing methods either limit to small motion or focus on topologically similar objects, leading to artifacts and inconsistency in the interpolation results. To overcome this challenge, we delve into pre-trained image diffusion models for their capabilities in semantic cognition and representations, ensuring consistent expression of the absent intermediate semantic representations with the input. To this end, we propose DreamMover, a novel image interpolation framework with three main components: 1) A natural flow estimator based on the diffusion model that can implicitly reason about the semantic correspondence between two images. 2) To avoid the loss of detailed information during fusion, our key insight is to fuse information in two parts, high-level space and low-level space. 3) To enhance the consistency between the generated images and input, we propose the self-attention concatenation and replacement approach. Lastly, we present a challenging benchmark dataset InterpBench to evaluate the semantic consistency of generated results. Extensive experiments demonstrate the effectiveness of our method. Our project is available at https://dreamm0ver.github.io .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑