Recently, pre-trained model and efficient parameter tuning have achieved remarkable success in natural language processing and high-level computer vision with the aid of masked modeling and prompt tuning. In low-level computer vision, however, there have been limited investigations on pre-trained models and even efficient fine-tuning strategy has not yet been explored despite its importance and benefit in various real-world tasks such as alleviating memory inflation issue when integrating new tasks on AI edge devices. Here, we propose a novel efficient parameter tuning approach dubbed contribution-based low-rank adaptation (CoLoRA) for multiple image restorations along with effective pre-training method with random order degradations (PROD). Unlike prior arts that tune all network parameters, our CoLoRA effectively fine-tunes small amount of parameters by leveraging LoRA (low-rank adaptation) for each new vision task with our contribution-based method to adaptively determine layer by layer capacity for that task to yield comparable performance to full tuning. Furthermore, our PROD strategy allows to extend the capability of pre-trained models with improved performance as well as robustness to bridge synthetic pre-training and real-world fine-tuning. Our CoLoRA with PROD has demonstrated its superior performance in various image restoration tasks across diverse degradation types on both synthetic and real-world datasets for known and novel tasks.
Continuous Expert Assembly: Instance-Conditioned Low-Rank Residuals for All-in-One Image Restoration
Southern University of Science and Technology(南方科技大学);Shenzhen University(深圳大学);Shenzhen University of Advanced Technology(深圳大学先进技术学院);Xi’an Jiaotong University(西安交通大学)
PromptHSI: Universal Hyperspectral Image Restoration with Vision-Language Modulated Frequency Adaptation
Institute of Data Science, National Cheng Kung University(成功大学数据科学研究所);Graduate Institute of Communication Engineering, National Taiwan University(台湾大学通讯工程学系);NVIDIA(英伟达公司)
From Rank Estimation to Rank Approximation: Rank Residual Constraint for Image Restoration
University of Electronic Science and Technology of China(电子科技大学);Nokia Bell Labs(诺基亚贝尔实验室);Nanyang Technological University(南洋理工大学);University of Macau(澳门大学);Nanjing Institute of Technology(南京工程学院)
Any Image Restoration via Efficient Spatial-Frequency Degradation Adaptation
University of Trento(特伦托大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Würzburg(乌尔姆大学);ETH Zürich(苏黎世联邦理工学院);Taiyuan University of Technology(太原理工大学);University of California, Merced(加州大学默塞德分校)
LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Weighted Low-rank Tensor Recovery for Hyperspectral Image Restoration
National Laboratory of Radar Signal Processing, Xidian University(西安电子科技大学雷达信号处理国家重点实验室);Science and Technology on Multispectral Information Processing Laboratory, School of Automation, Huazhong University of Science and Technology(华中科技大学自动化学院多谱信息处理技术国家级重点实验室)
Non-local Meets Global: An Iterative Paradigm for Hyperspectral Image Restoration
RIKEN Center for Advanced Intelligence Project (AIP)(理化学研究所先进智能项目中心);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);Paradigm Inc.(第四范式有限公司);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院);LISMARS, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室)
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Yunnan, China(昆明理工大学信息工程与自动化学院,云南,中国);School of Mathematics and Statistics, Yunnan University, Yunnan, China(云南大学数学与统计学院,云南,中国);Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology, Shenzhen, China(深圳视觉对象检测与识别重点实验室,哈尔滨工业大学,深圳,中国)
Deep LoRA-Unfolding Networks for Image Restoration
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(计算机科学与技术学院,哈尔滨工业大学(深圳));Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(图像通信与网络工程院,上海交通大学);School of Intelligence Science and Technology, Nanjing University(智能科学与技术学院,南京大学);School of Electronics and Information Engineering, Harbin Institute of Technology (Shenzhen)(电子与信息工程学院,哈尔滨工业大学(深圳))
Cooperative Semantic Segmentation and Image Restoration in Adverse Environmental Conditions
Tsinghua University(清华大学);Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy of Science(中国科学院深圳先进技术研究院);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);University College London(伦敦大学学院)
CoRE-UIR: Prior-guided common and residual experts for efficient all-in-one remote sensing image restoration
School of Geodesy and Geomatics, Wuhan University(武汉大学测绘学院);School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University(武汉大学测绘遥感信息工程国家重点实验室)
A Unified Foundation Model for All-in-One Multi-Modal Remote Sensing Image Restoration and Fusion with Language Prompting
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences(中国科学院大学电子电气与通信工程学院)
HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Mathematics and Statistics, Xi’an Jiaotong University(西安交通大学数学与统计学院)
Temporal Restoration and Spatial Rewiring for Source-Free Multivariate Time Series Domain Adaptation
Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Institute for Infocomm Research, Agency for Science Technology and Research (A*STAR)(信息通信研究院,科技研究局(A*STAR));Nanyang Technological University(南洋理工大学);Centre for Frontier AI Research, Agency for Science Technology and Research (A*STAR)(前沿人工智能研究中心,科技研究局(A*STAR))
Local Margin Restoration for Test-Time Adaptation of Vision-Language Models
Guangzhou University(广州大学);The Second Affiliated Hospital of Guangzhou University of Chinese Medicine(广州中医药大学第二附属医院);Jinan University(暨南大学);Pengcheng Laboratory(鹏城实验室)
FADRA: Frequency-Aware Diffusion with Residual Adaptation for Video Face Restoration
United Arab Emirates University (UAEU)(阿联酋大学);Ocean University of China (OUC)(中国海洋大学);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)