Continual learning is an emerging topic in the field of deep learning, where a model is expected to learn continuously for new upcoming tasks without forgetting previous experiences. This field has witnessed numerous advancements, but few works have been attempted in the direction of image restoration. Handling large image sizes and the divergent nature of various degradation poses a unique challenge in the restoration domain. However, existing works require heavily engineered architectural modifications for new task adaptation, resulting in significant computational overhead. Regularization-based methods are unsuitable for restoration, as different restoration challenges require different kinds of feature processing. In this direction, we propose a simple modification of the convolution layer to adapt the knowledge from previous restoration tasks without touching the main backbone architecture. Therefore, it can be seamlessly applied to any deep architecture without any structural modifications. Unlike other approaches, we demonstrate that our model can increase the number of trainable parameters without significantly increasing computational overhead or inference time. Experimental validation demonstrates that new restoration tasks can be introduced without compromising the performance of existing tasks. We also show that performance on new restoration tasks improves by adapting the knowledge from the knowledge base created by previous restoration tasks. The code is available at https://github.com/aupendu/continual-restore.
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；The Chinese University of Hong Kong（香港中文大学）
In image restoration tasks, like denoising and super resolution, continual modulation of restoration levels is of great importance for real-world applications, but has failed most of existing deep learning based image restoration methods. Learning from discrete and fixed restoration levels, deep models cannot be easily generalized to data of continuous and unseen levels. This topic is rarely touched in literature, due to the difficulty of modulating well-trained models with certain hyper-parameters. We make a step forward by proposing a unified CNN framework that consists of few additional par…
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Yunnan, China（昆明理工大学信息工程与自动化学院，云南，中国）；School of Mathematics and Statistics, Yunnan University, Yunnan, China（云南大学数学与统计学院，云南，中国）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology, Shenzhen, China（深圳视觉对象检测与识别重点实验室，哈尔滨工业大学，深圳，中国）
Sharing Key Semantics in Transformer Makes Efficient Image Restoration
ETH Zürich（苏黎世联邦理工学院）；INSAIT, Sofia University（保加利亚索菲亚大学INSAIT研究院）；University of Pisa（比萨大学）；University of Trento（特伦托大学）；Meta Reality Labs（元宇宙现实实验室）；Peking University（北京大学）；University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；University of California, Merced（加州大学默塞德分校）
Recently, much attention has been spent on neural architecture search (NAS), aiming to outperform those manually-designed neural architectures on high-level vision recognition tasks. Inspired by the success, here we attempt to leverage NAS techniques to automatically design efficient network architectures for low-level image restoration tasks. In particular, we propose a memory-efficient hierarchical NAS (termed HiNAS) and apply it to two such tasks: image denoising and image super-resolution. HiNAS adopts gradient based search strategies and builds a flexible hierarchical search space, includ…
CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China（中国科学技术大学中国科学院空间信息处理与应用系统技术重点实验室）
Most existing image restoration networks are designed in a disposable way and catastrophically forget previously learned distortions when trained on a new distortion removal task. To alleviate this problem, we raise the novel lifelong image restoration problem for blended distortions. We first design a base fork-join model in which multiple pre-trained expert models specializing in individual distortion removal task work cooperatively and adaptively to handle blended distortions. When the input is degraded by a new distortion, inspired by adult neurogenesis in human memory system, we develop a…
Diverse user preferences over images have recently led to a great amount of interest in controlling the imagery effects for image restoration tasks. However, existing methods require separate inference through the entire network per each output, which hinders users from readily comparing multiple imagery effects due to long latency. To this end, we propose a novel framework based on a neural architecture search technique that enables efficient generation of multiple imagery effects via two stages of pruning: task-agnostic and task-specific pruning. Specifically, task-specific pruning learns to…
National University of Defense Technology（国防科技大学）；National Innovation Institute of Defense Technology（国防科技创新研究院）
The great success of Deep Neural Networks (DNNs) has inspired the algorithmic development of DNN-based Fixed-Point (DNN-FP) for computer vision tasks. DNN-FP methods, trained by Back-Propagation Through Time or computing the inaccurate inversion of the Jacobian, suffer from inferior representation ability. Motivated by the representation power of the Transformer, we propose a framework to unroll the FP and approximate each unrolled process via Transformer blocks, called FPformer. To reduce the high consumption of memory and computation, we come up with FPRformer by sharing parameters between t…
We present the Cross-stitched Multi-task Unified Dual Recursive Network (CMUDRN) model targeting the task of unified deraining and desnowing in a multi-task learning setting. This unified model borrows from the basic Dual Recursive Network (DRN) architecture developed by Cai et al. The proposed model makes use of cross-stitch units that enable multi-task learning across two separate DRN models, each tasked for single image deraining and desnowing, respectively. By fixing cross-stitch units at several layers of basic task-specific DRN networks, we perform multi-task learning over the two separa…
Tianjin University（天津大学）；Australian National University（澳大利亚国立大学）；Agency for Science, Technology and Research（新加坡科技研究局）
Images captured under outdoor scenes usually suffer from low contrast and limited visibility due to suspended atmospheric particles, which directly affects the quality of photos. Despite numerous image dehazing methods have been proposed, effective hazy image restoration remains a challenging problem. Existing learning-based methods usually predict the medium transmission by Convolutional Neural Networks (CNNs), but ignore the key global atmospheric light. Different from previous learning-based methods, we propose a flexible cascaded CNN for single hazy image restoration, which considers the m…
Southern University of Science and Technology（南方科技大学）；A*STAR（新加坡科技研究局）；New York University（纽约大学）；Chinese Academy of Sciences（中国科学院）；Shenzhen People’s Hospital（深圳市人民医院）
Fundus photography is a routine examination in clinics to diagnose and monitor ocular diseases. However, for cataract patients, the fundus image always suffers quality degradation caused by the clouding lens. The degradation prevents reliable diagnosis by ophthalmologists or computer-aided systems. To improve the certainty in clinical diagnosis, restoration algorithms have been proposed to enhance the quality of fundus images. Unfortunately, challenges remain in the deployment of these algorithms, such as collecting sufficient training data and preserving retinal structures. In this paper, to …
Continual Test-Time Adaptation for Object Detection with Adaptive Monitoring and Randomized Restoration
School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学与技术学院）；State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University（清华大学智能技术与系统国家重点实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生学院）；National Supercomputing Center in Shenzhen（深圳国家超算中心）；Ministry of Education Key Laboratory for Earth System Modeling and the Department of Earth System Science, Tsinghua University（清华大学地球系统模型教育部重点实验室）
Dynamic Pre-training: Towards Efficient and Scalable All-in-One Image Restoration
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Inception Institute of AI（Inception人工智能研究院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；University of California, Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google Research（谷歌研究院）
AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Robotics, Wuhan University（武汉大学机器人学院）；Suzhou Institute, Wuhan University（武汉大学苏州研究院）
UCAN: Unified Convolutional Attention Network for Expansive Receptive Fields in Lightweight Super-Resolution
Ho Chi Minh City Open University（胡志明市开放大学）；AI Tech Lab, Ho Chi Minh City University of Technology（胡志明市科技大学人工智能技术实验室）；Code Mely AI Research Team（Code Mely 人工智能研究团队）；Global College, Shanghai Jiao Tong University（上海交通大学全球学院）；Ha Noi University of Science and Technology（河内科技大学）；University of Manitoba（曼尼托巴大学）
FedCTTA: A Collaborative Approach to Continual Test-Time Adaptation in Federated Learning
1 Center for Computational \& Data Sciences, Independent University, Bangladesh；2 Curtin University；Center for Computational \& Data Sciences Independent University, Bangladesh；School of Electrical Engineering, Computing \& Mathematical Sciences (EECMS) Curtin University
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
Department of XXX, University of YYY, Location, Country（XXX部门，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）；State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi'an, China（信息服务网络国家重点实验室，电信工程学院，西安电子科技大学，西安，中国）；School of Electronic Engineering, Xidian University, Xi'an, China（电子工程学院，西安电子科技大学，西安，中国）
Continual All-in-One Adverse Weather Removal with Knowledge Replay on a Unified Network Structure
School of Telecommunications Engineering, Xidian University（西安电子科技大学通信工程学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；University of New South Wales (UNSW)（新南威尔士大学）；Shandong Normal University（山东师范大学）
Humans learn adaptively and efficiently throughout their lives. However, incrementally learning tasks causes artificial neural networks to overwrite relevant information learned about older tasks, resulting in 'Catastrophic Forgetting'. Efforts to overcome this phenomenon often utilize resources poorly, for instance, by growing the network architecture or needing to save parametric importance scores, or violate data privacy between tasks. To tackle this, we propose SPACE, an algorithm that enables a network to learn continually and efficiently by partitioning the learnt space into a Core space…
Continual Test-Time Adaptation in Computer Vision: Methods, Benchmarks, and Future Directions
The University of Texas at Dallas（德克萨斯大学达拉斯分校）；LIVIA ETS Montreal, ILLS International Laboratory on Learning Systems (ILLS)（蒙特利尔LIVIA ETS，学习系统国际实验室（ILLS））；Tulane University（路易斯安那州立大学）；Nanyang Technological University（南洋理工大学）；Hanyang University（翰阳大学）
iGSP:Implicit Gradient Subspace Projection for Efficient Continual Learning of Vision-Language Models
School of Geosciences and Info-Physics, Central South University（地质科学与信息物理学院，中南大学）；School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology（地球科学与空间信息工程学院，湖南科技大学）
Zero-Shot Video Restoration and Enhancement Using Pre-Trained Image Diffusion Model
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；School of Engineering Science, Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学工程科学学院）
From Redaction to Restoration: Deep Learning for Medical Image Anonymization and Reconstruction
Center for Artificial Intelligence, BCVI, Northwestern Medicine（西北医学BCVI人工智能中心）；Department of Electrical and Computer Engineering, Northwestern University（西北大学电气与计算机工程系）；Department of Surgery, Northwestern University（西北大学外科学系）；Xtasis Inc.（Xtasis公司）；Medtronic（美敦力公司）；Departments of Radiology and Clinical Neurosciences, University of Calgary（卡尔加里大学放射学与临床神经科学系）
College of Electric and Information, Northeast Agricultural University（东北农业大学电气与信息学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
