UniE2F: A Unified Diffusion Framework for Event-to-Frame Reconstruction with Video Foundation Models
Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Department of Computer Science, City University of Hong Kong (Dongguan)(香港城市大学(东莞)计算机科学系)
Gang Xu, Zhiyu Zhu, Junhui Hou
备注:
This paper has been accepted by IEEE Transactions on Image Processing (TIP)
Event cameras excel at high-speed, low-power, and high-dynamic-range scene perception. However, as they fundamentally record only relative intensity changes rather than absolute intensity, the resulting data streams suffer from a significant loss of spatial information and static texture details. In this paper, we address this limitation by leveraging the generative prior of a pre-trained video diffusion model to reconstruct high-fidelity video frames from sparse event data. Specifically, we first establish a baseline model by directly applying event data as a condition to synthesize videos. Then, based on the physical correlation between the event stream and video frames, we further introduce the event-based inter-frame residual guidance to enhance the accuracy of video frame reconstruction. Furthermore, we extend our method to video frame interpolation and prediction in a zero-shot manner by modulating the reverse diffusion sampling process, thereby creating a unified event-to-frame reconstruction framework. Experimental results on real-world and synthetic datasets demonstrate that our method outperforms previous reconstruction approaches on most quantitative metrics and achieves superior qualitative results, while achieving competitive zero-shot performance compared with dedicated methods explicitly trained for interpolation and prediction tasks. We also refer the reviewers to the video demo contained in the supplementary material for video results. The code will be publicly available at https://github.com/CS-GangXu/UniE2F.
VideoRAE: Taming Video Foundation Models for Generative Modeling via Representation Autoencoders
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Huazhong University of Science and Technology(华中科技大学);Shenzhen Loop Area Institute(深圳河套学院);University of Science and Technology of China(中国科学技术大学)
Beyond Training from Scratch: Foundation Models for Data-Efficient and Generalizable Cardiac MRI Reconstruction
Research Ireland Centre for Research Training in Machine Learning(爱尔兰研究机器学习研究中心);Dublin City University(都柏林城市大学);University College Dublin(都柏林大学学院);Rinn Artificial Intelligence(Rinn人工智能公司)
Online3R: Online Learning for Consistent Sequential Reconstruction Based on Geometry Foundation Model
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);T-Stone Robotics Institute, The Chinese University of Hong Kong(香港中文大学天石机器人研究所);NVIDIA(英伟达);College of Computer and Information Science, Southwest University(西南大学计算机与信息科学学院);School of Artificial Intelligence and Computer Science, Anqing Normal University(安庆师范大学人工智能与计算机科学学院)
On the Utility of Foundation Models for Fast MRI: Vision-Language-Guided Image Reconstruction
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital(麻省总医院Athinoula A. Martinos生物医学成像中心);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院)
Regist3R: Incremental Registration with Stereo Foundation Model
College of Computer Science and Technology(计算机科学与技术学院);National Key Laboratory of Parallel and Distributed Computing(并行与分布式计算国家重点实验室);National University of Defense Technology(国防科技大学)
GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priors
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Peking University(北京大学);The Hong Kong University of Science and Technology(香港科技大学)
DeepSparse: A Foundation Model for Sparse-View CBCT Reconstruction
Department of Electronic and Computer Engineering, the Hong Kong University of Science and Technology(电子与计算机工程系,香港科学与技术大学);School of Cyber Science and Engineering, Sichuan University(网络科学与工程学院,四川大学)
CRUNet-MR-Univ: A Foundation Model for Diverse Cardiac MRI Reconstruction
Department of Radiology, Leiden University Medical Center, Leiden, The Netherlands(莱顿大学医学中心放射科);Philips Innovative Technologies, Hamburg, Germany(飞利浦创新技术)
Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video
School of Computer Science and Engineering, Sun Yat-sen University, China(中山大学计算机科学与工程学院);Tsinghua Shenzhen International Graduate School, Tsinghua University, China(清华大学深圳国际研究生院)
EndoSfM3D: Learning to 3D Reconstruct Any Endoscopic Surgery Scene using Self-supervised Foundation Model
UCL Hawkes Institute(UCL哈维斯研究所);Department of Medical Physics(医学物理系);Biomedical Engineering, University College London, UK(生物医学工程,伦敦大学学院,英国);Division of Informatics, Imaging and Data Sciences, The University of Manchester, Manchester, UK(信息学、成像与数据科学部门,曼彻斯特大学,曼彻斯特,英国)
Adaptive World Memory 3D Foundation Model for Scalable 3D Mapping, Localization, and Rendering
Shanghai Jiao Tong University(上海交通大学);State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis(航空电子综合与航空系统体系综合国家重点实验室);Shanghai Key Laboratory of Navigation and Location Based Services(上海市导航与位置服务重点实验室);Nanyang Technological University(南洋理工大学);University of Technology Nuremberg(纽伦堡工业大学)
EO-VGGT: Orbital Ray-Conditioned 3D Foundation Models for Satellite Multi-View Reconstruction
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室);Hubei Luojia Laboratory(湖北珞珈实验室)