In this paper, we propose the first diffusion-based all-in-one video restoration method that utilizes the power of a pre-trained Stable Diffusion and a fine-tuned ControlNet. Our method can restore various types of video degradation with a single unified model, overcoming the limitation of standard methods that require specific models for each restoration task. Our contributions include an efficient training strategy with Task Prompt Guidance (TPG) for diverse restoration tasks, an inference strategy that combines Denoising Diffusion Implicit Models~(DDIM) inversion with a novel Sliding Window Cross-Frame Attention (SW-CFA) mechanism for enhanced content preservation and temporal consistency, and a scalable pipeline that makes our method all-in-one to adapt to different video restoration tasks. Through extensive experiments on five video restoration tasks, we demonstrate the superiority of our method in generalization capability to real-world videos and temporal consistency preservation over existing state-of-the-art methods. Our method advances the video restoration task by providing a unified solution that enhances video quality across multiple applications.
University of Ontario Institute of Technology（安大略理工大学）
Greyscale image colorization for applications in image restoration has seen significant improvements in recent years. Many of these techniques that use learning-based methods struggle to effectively colorize sparse inputs. With the consistent growth of the anime industry, the ability to colorize sparse input such as line art can reduce significant cost and redundant work for production studios by eliminating the in-between frame colorization process. Simply using existing methods yields inconsistent colors between related frames resulting in a flicker effect in the final video. In order to suc…
Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References
School of Electrical and Information Engineering, Tianjin University（天津大学电气自动化与信息工程学院）；Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学（LUT））
FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restoration
United Arab Emirates University (UAEU)（阿联酋大学）；Ocean University of China (OUC)（中国海洋大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；School of Engineering Science, Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学工程科学学院）
Physics-Guided Null-Space Diffusion with Sparse Masking for Corrective Sparse-View CT Reconstruction
School of Information Engineering, Nanchang University, Nanchang, China（信息工程学院，南昌大学，南昌，中国）；School of Mathematics and Computer Sciences, Nanchang University, Nanchang, China（数学与计算机科学学院，南昌大学，南昌，中国）
Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Video restoration tasks, including super-resolution, deblurring, etc, are drawing increasing attention in the computer vision community. A challenging benchmark named REDS is released in the NTIRE19 Challenge. This new benchmark challenges existing methods from two aspects: (1) how to align multiple frames given large motions, and (2) how to effectively fuse different frames with diverse motion and blur. In this work, we propose a novel Video Restoration framework with Enhanced Deformable networks, termed EDVR, to address these challenges. First, to handle large motions, we devise a Pyramid, C…
Temporal Restoration and Spatial Rewiring for Source-Free Multivariate Time Series Domain Adaptation
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Institute for Infocomm Research, Agency for Science Technology and Research (A*STAR)（信息通信研究院，科技研究局（A*STAR））；Nanyang Technological University（南洋理工大学）；Centre for Frontier AI Research, Agency for Science Technology and Research (A*STAR)（前沿人工智能研究中心，科技研究局（A*STAR））
All-in-One Video Restoration under Smoothly Evolving Unknown Weather Degradations
Department of Computer Science and Technology, Harbin Institute of Technology（计算机科学与技术系，哈尔滨工业大学）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）；School of Computer Science, Peking University（计算机学院，北京大学）
MiVID: Multi-Strategic Self-Supervision for Video Frame Interpolation using Diffusion Model
School of Computer Engineering, KIIT Deemed to be University（计算机工程学院，KIIT被认定大学）；Department of Computer Science and Engineering, National Institute of Technology（计算机科学与工程系，国家理工学院）
