Diverse user preferences over images have recently led to a great amount of interest in controlling the imagery effects for image restoration tasks. However, existing methods require separate inference through the entire network per each output, which hinders users from readily comparing multiple imagery effects due to long latency. To this end, we propose a novel framework based on a neural architecture search technique that enables efficient generation of multiple imagery effects via two stages of pruning: task-agnostic and task-specific pruning. Specifically, task-specific pruning learns to adaptively remove the irrelevant network parameters for each task, while task-agnostic pruning learns to find an efficient architecture by sharing the early layers of the network across different tasks. Since the shared layers allow for feature reuse, only a single inference of the task-agnostic layers is needed to generate multiple imagery effects from the input image. Using the proposed task-agnostic and task-specific pruning schemes together significantly reduces the FLOPs and the actual latency of inference compared to the baseline. We reduce 95.7% of the FLOPs when generating 27 imagery effects, and make the GPU latency 73.0% faster on 4K-resolution images.
Tsinghua University（清华大学）；Chinese University of Hong Kong（香港中文大学）；University of Rochester（罗切斯特大学）
Deep learning methods have witnessed the great progress in image restoration with specific metrics (e.g., PSNR, SSIM). However, the perceptual quality of the restored image is relatively subjective, and it is necessary for users to control the reconstruction result according to personal preferences or image characteristics, which cannot be done using existing deterministic networks. This motivates us to exquisitely design a unified interactive framework for general image restoration tasks. Under this framework, users can control continuous transition of different objectives, e.g., the percepti…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Interactive image restoration aims to generate restored images by adjusting a controlling coefficient which determines the restoration level. Previous works are restricted in modulating image with a single coefficient. However, real images always contain multiple types of degradation, which cannot be well determined by one coefficient. To make a step forward, this paper presents a new problem setup, called multi-dimension (MD) modulation, which aims at modulating output effects across multiple degradation types and levels. Compared with the previous single-dimension (SD) modulation, the MD is …
HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restoration
Hawkes Institute, UCL（UCL哈维斯研究所）；Dept. of Medical Physics and Biomedical Engineering, UCL（UCL医学物理与生物医学工程系）；Dept. of Computer Science, UCL（UCL计算机科学系）；Centre for AI, DS&AI, AstraZeneca, UK（阿斯利康AI中心）
Existing low-rank tensor completion (LRTC) approaches aim at restoring a partially observed tensor by imposing a global low-rank constraint on the underlying completed tensor. However, such a global rank assumption suffers the trade-off between restoring the originally details-lacking parts and neglecting the potentially complex objects, making the completion performance unsatisfactory on both sides. To address this problem, we propose a novel and practical strategy for image restoration that restores the partially observed tensor in a coarse-to-fine (C2F) manner, which gets rid of such trade-…
Recently, much attention has been spent on neural architecture search (NAS), aiming to outperform those manually-designed neural architectures on high-level vision recognition tasks. Inspired by the success, here we attempt to leverage NAS techniques to automatically design efficient network architectures for low-level image restoration tasks. In particular, we propose a memory-efficient hierarchical NAS (termed HiNAS) and apply it to two such tasks: image denoising and image super-resolution. HiNAS adopts gradient based search strategies and builds a flexible hierarchical search space, includ…
Modulating image restoration level aims to generate a restored image by altering a factor that represents the restoration strength. Previous works mainly focused on optimizing the mean squared reconstruction error, which brings high reconstruction accuracy but lacks finer texture details. This paper presents a Controllable Unet Generative Adversarial Network (CUGAN) to generate high-frequency textures in the modulation tasks. CUGAN consists of two modules -- base networks and condition networks. The base networks comprise a generator and a discriminator. In the generator, we realize the intera…
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Yunnan, China（昆明理工大学信息工程与自动化学院，云南，中国）；School of Mathematics and Statistics, Yunnan University, Yunnan, China（云南大学数学与统计学院，云南，中国）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology, Shenzhen, China（深圳视觉对象检测与识别重点实验室，哈尔滨工业大学，深圳，中国）
SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学教育部人工智能重点实验室、人工智能学院）
Jiangxi Normal University（江西师范大学）；University of Science and Technology of China（中国科学技术大学）；East China Normal University（华东师范大学）；Beijing Jiaotong University（北京交通大学）
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Image denoising is a classic restoration problem. Yet, current deep learning methods are subject to the problems of generalization and interpretability. To mitigate these problems, in this project, we present a framework that is capable of controllable, confidence-based noise removal. The framework is based on the fusion between two different denoised images, both derived from the same noisy input. One of the two is denoised using generic algorithms (e.g. Gaussian), which make few assumptions on the input images, therefore, generalize in all scenarios. The other is denoised using deep learning…
Towards Controllable Low-Light Image Enhancement: A Continuous Multi-illumination Dataset and Efficient State Space Framework
Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；School of Automation, China University of Geosciences（中国地质大学自动化学院）；Department of Electromechanical Engineering, University of Macau（澳门大学机电工程系）
ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts
Electronic Information School, Wuhan University（武汉大学电子信息学院）；School of Computer Science and Engineering, Macau University of Science and Technology（澳门科学技术大学计算机科学与工程学院）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；State Key Lab of Intelligent Transportation System, Beihang University（北航智能交通系统国家重点实验室）
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Harbin Institute of Technology（哈尔滨工业大学）
Non-local self-similarity in natural images has been well studied as an effective prior in image restoration. However, for single image super-resolution (SISR), most existing deep non-local methods (e.g., non-local neural networks) only exploit similar patches within the same scale of the low-resolution (LR) input image. Consequently, the restoration is limited to using the same-scale information while neglecting potential high-resolution (HR) cues from other scales. In this paper, we explore the cross-scale patch recurrence property of a natural image, i.e., similar patches tend to recur many…
Peking University Shenzhen Graduate School（北京大学深圳研究生院）；Peng Cheng Laboratory（鹏城实验室）；Tencent PCG（腾讯平台与内容事业群）；Shanghai AI Laboratory（上海人工智能实验室）
Interactive image restoration aims to restore images by adjusting several controlling coefficients, which determine the restoration strength. Existing methods are restricted in learning the controllable functions under the supervision of known degradation types and levels. They usually suffer from a severe performance drop when the real degradation is different from their assumptions. Such a limitation is due to the complexity of real-world degradations, which can not provide explicit supervision to the interactive modulation during training. However, how to realize the interactive modulation …
Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Electronic Information School, Wuhan University（武汉大学电子信息学院）
With the recently massive development in convolution neural networks, numerous lightweight CNN-based image super-resolution methods have been proposed for practical deployments on edge devices. However, most existing methods focus on one specific aspect: network or loss design, which leads to the difficulty of minimizing the model size. To address the issue, we conclude block devising, architecture searching, and loss design to obtain a more efficient SR structure. In this paper, we proposed an edge-enhanced feature distillation network, named EFDN, to preserve the high-frequency information u…
Remote Sensing Image Dehazing: A Systematic Review of Progress, Challenges, and Prospects
School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院）；Josef Kittler Research Institute on Artificial Intelligence（乔塞夫·基特勒人工智能研究所）；School of Electronic Engineering（电子工程学院）；College of Artificial Intelligence（人工智能学院）；Aerospace Information Research Institute（航天信息研究所）；Department of Computer Science（计算机科学系）
Hangzhou Dianzi University（杭州电子科技大学）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；University of Science and Technology of China（中国科学技术大学）
The depth images denoising are increasingly becoming the hot research topic nowadays because they reflect the three-dimensional (3D) scene and can be applied in various fields of computer vision. But the depth images obtained from depth camera usually contain stains such as noise, which greatly impairs the performance of depth related applications. In this paper, considering that group-based image restoration methods are more effective in gathering the similarity among patches, a group based nuclear norm and learning graph (GNNLG) model was proposed. For each patch, we find and group the most …
Joint Degradation-Aware Arbitrary-Scale Super-Resolution for Variable-Rate Extreme Image Compression
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University（上海交通大学信息科学与电子工程学院）；Department of Electronic Engineer and Information Science, University of Science and Technology of China（中国科学技术大学电子工程与信息科学系）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能教育部重点实验室）
Zero-shot capability has been considered as a new revolution of deep learning, letting machines work on tasks without curated training data. As a good start and the only existing outcome of zero-shot image captioning (IC), ZeroCap abandons supervised training and sequentially searches every word in the caption using the knowledge of large-scale pretrained models. Though effective, its autoregressive generation and gradient-directed searching mechanism limit the diversity of captions and inference speed, respectively. Moreover, ZeroCap does not consider the controllability issue of zero-shot IC…
CodeEnhance: A Codebook-Driven Approach for Low-Light Image Enhancement
Computer Vision Institute, College of Computer Science and Software Engineering, Shenzhen University（计算机视觉研究所，计算机科学与软件工程学院，深圳大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（社会人工智能与机器人研究所）；Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University（广东省智能信息处理重点实验室，深圳大学）；School of AI and Advanced Computing, Xi’an Jiaotong-Liverpool University（人工智能与高级计算学院，西安交通大学-利物浦大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）；Department of Electrical & Computer Engineering, University of Alberta（电气与计算机工程系，阿尔伯塔大学）
VinAI Research（VinAI研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Posts & Telecommunications Institute of Technology（邮政电信技术学院）
The introduction of high-quality image generation models, particularly the StyleGAN family, provides a powerful tool to synthesize and manipulate images. However, existing models are built upon high-quality (HQ) data as desired outputs, making them unfit for in-the-wild low-quality (LQ) images, which are common inputs for manipulation. In this work, we bridge this gap by proposing a novel GAN structure that allows for generating images with controllable quality. The network can synthesize various image degradation and restore the sharp image via a quality control code. Our proposed QC-StyleGAN…
A Fast and Precise Method for Searching Rectangular Tumor Regions in Brain MR Images
Software Technologies Group, MRI Systems Development Department, MRI Systems Division, Canon Medical Systems Corporation, Tochigi, Japan（Canon医疗系统公司软件技术组、MRI系统开发部、MRI系统事业部，Tochigi）
LUCID: Learning Unified Control for Image Deflaring and Exposure Mastery in Nighttime Photography
MoE Key Lab of Artificial Intelligence（人工智能混合专家实验室）；AI Institute（人工智能研究所）；School of Computer Science（计算机科学学院）；School of Biomedical Engineering（生物医学工程学院）；School of Artificial Intelligence（人工智能学院）
PairingNet: A Learning-based Pair-searching and -matching Network for Image Fragments
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；The University of Tokyo（东京大学）；National University of Defense Technology（国防科技大学）；School of Archaeology, Jilin University（吉林大学考古学院）
