We study the problem of generating intermediate images from image pairs with large motion while maintaining semantic consistency. Due to the large motion, the intermediate semantic information may be absent in input images. Existing methods either limit to small motion or focus on topologically similar objects, leading to artifacts and inconsistency in the interpolation results. To overcome this challenge, we delve into pre-trained image diffusion models for their capabilities in semantic cognition and representations, ensuring consistent expression of the absent intermediate semantic representations with the input. To this end, we propose DreamMover, a novel image interpolation framework with three main components: 1) A natural flow estimator based on the diffusion model that can implicitly reason about the semantic correspondence between two images. 2) To avoid the loss of detailed information during fusion, our key insight is to fuse information in two parts, high-level space and low-level space. 3) To enhance the consistency between the generated images and input, we propose the self-attention concatenation and replacement approach. Lastly, we present a challenging benchmark dataset InterpBench to evaluate the semantic consistency of generated results. Extensive experiments demonstrate the effectiveness of our method. Our project is available at https://dreamm0ver.github.io .
Envision3D: One Image to 3D with Anchor Views Interpolation
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);National University of Singapore(新加坡国立大学);PengCheng Laboratory(鹏城实验室);Rabbitpre(睿比特科技)
EventDiff: A Unified and Efficient Diffusion Model Framework for Event-based Video Frame Interpolation
Center for Brain Inspired Computing Research (CBICR), Department of Precision Instrument, Tsinghua University(脑启发计算研究中心(CBICR)、精密仪器系,清华大学);College of Computer Science and Technology, Taiyuan University of Technology(计算机科学与技术学院,太原理工大学);The Information Science Academy of China Electronics Technology Group Corporation(中国电子科技集团有限公司信息科学学院);State Key Laboratory of Integrated Chips and Systems, School of Information Science and Technology, Fudan University(集成电路与系统集成国家重点实验室,复旦大学信息科学学院);Engineering Laboratory of Power Equipment Reliability in Complicated Coastal Environments, Tsinghua Shenzhen International Graduate School, Tsinghua University(复杂近海环境电力设备可靠性工程实验室,清华大学深圳国际研究生院)
UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models
Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系)
Unrolling Graph-based Douglas-Rachford Algorithm for Image Interpolation with Informed Initialization
Shandong Provincial Key Laboratory of Smart Mine Information Technology, China(山东省智慧矿山信息技术重点实验室);Qingdao Key Laboratory of Intelligent Sensing and Virtual Emergency Simulation, China(青岛市智能感知与虚拟应急仿真重点实验室);College of Computer Science and Engineering, Shandong University of Science and Technology, China(山东科技大学计算机科学与工程学院);Dept of EECS, York University, Canada(约克大学电气工程与计算机科学系)
Continuous sPatial-Temporal Deformable Image Registration (CPT-DIR) for motion modelling in radiotherapy: beyond classic voxel-based methods
Center for Proton Therapy, Paul Scherrer Institut(保罗·施特格研究所质子治疗中心);Department of Computer Science, ETH Zürich(苏黎世联邦理工学院计算机科学系);Department of Physics, ETH Zürich(苏黎世联邦理工学院物理系);Department of Automation, Tsinghua Univeristy(清华大学自动化系);S-Lab, Nanyang Technological University(南洋理工大学S实验室)