HighEr-Resolution Network for Image Demosaicing and Enhancing
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；East China Normal University（华东师范大学）；Kuaishou Technology（快手科技）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Neural-networks based image restoration methods tend to use low-resolution image patches for training. Although higher-resolution image patches can provide more global information, state-of-the-art methods cannot utilize them due to their huge GPU memory usage, as well as the instable training process. However, plenty of studies have shown that global information is crucial for image restoration tasks like image demosaicing and enhancing. In this work, we propose a HighEr-Resolution Network (HERN) to fully learning global information in high-resolution image patches. To achieve this, the HERN employs two parallel paths to learn image features in two different resolutions, respectively. By combining global-aware features and multi-scale features, our HERN is able to learn global information with feasible GPU memory usage. Besides, we introduce a progressive training method to solve the instability issue and accelerate model convergence. On the task of image demosaicing and enhancing, our HERN achieves state-of-the-art performance on the AIM2019 RAW to RGB mapping challenge. The source code of our implementation is available at https://github.com/MKFMIKU/RAW2RGBNet.
PIDSR: Complementary Polarized Image Demosaicing and Super-Resolution
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；National Institute of Informatics（国家信息研究所）；State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机科学学院多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学计算机科学学院视觉技术国家工程研究中心）
In digital photography, two image restoration tasks have been studied extensively and resolved independently: demosaicing and super-resolution. Both these tasks are related to resolution limitations of the camera. Performing super-resolution on a demosaiced images simply exacerbates the artifacts introduced by demosaicing. In this paper, we show that such accumulation of errors can be easily averted by jointly performing demosaicing and super-resolution. To this end, we propose a deep residual network for learning an end-to-end mapping between Bayer images and high-resolution images. By traini…
West Virginia University（西弗吉尼亚大学）；Case Western Reserve University（凯斯西储大学）
Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this paper, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supporte…
Image demosaicing is problem of interpolating full- resolution color images from raw sensor (color filter array) data. During last decade, deep neural networks have been widely used in image restoration, and in particular, in demosaicing, attaining significant performance improvement. In recent years, vision transformers have been designed and successfully used in various computer vision applications. One of the recent methods of image restoration based on a Swin Transformer (ST), SwinIR, demonstrates state-of-the-art performance with a smaller number of parameters than neural network-based me…
KAUST（阿卜杜拉国王科技大学）；The University of Sydney（悉尼大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；SenseTime Research（商汤科技研究院）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Imaging is usually a mixture problem of incomplete color sampling, noise degradation, and limited resolution. This mixture problem is typically solved by a sequential solution that applies demosaicing (DM), denoising (DN), and super-resolution (SR) sequentially in a fixed and predefined pipeline (execution order of tasks), DM$\to$DN$\to$SR. The most recent work on image processing focuses on developing more sophisticated architectures to achieve higher image quality. Little attention has been paid to the design of the pipeline, and it is still not clear how significant the pipeline is to image…
Compared to DSLR cameras, smartphone cameras have smaller sensors, which limits their spatial resolution; smaller apertures, which limits their light gathering ability; and smaller pixels, which reduces their signal-to noise ratio. The use of color filter arrays (CFAs) requires demosaicing, which further degrades resolution. In this paper, we supplant the use of traditional demosaicing in single-frame and burst photography pipelines with a multiframe super-resolution algorithm that creates a complete RGB image directly from a burst of CFA raw images. We harness natural hand tremor, typical in …
The Rapid and Accurate Image Super Resolution (RAISR) method of Romano, Isidoro, and Milanfar is a computationally efficient image upscaling method using a trained set of filters. We describe a generalization of RAISR, which we name Best Linear Adaptive Enhancement (BLADE). This approach is a trainable edge-adaptive filtering framework that is general, simple, computationally efficient, and useful for a wide range of problems in computational photography. We show applications to operations which may appear in a camera pipeline including denoising, demosaicing, and stylization.
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；Humboldt-University（洪堡大学）
Spectral imaging technologies have rapidly evolved during the past decades. The recent development of single-camera-one-shot techniques for hyperspectral imaging allows multiple spectral bands to be captured simultaneously (3x3, 4x4 or 5x5 mosaic), opening up a wide range of applications. Examples include intraoperative imaging, agricultural field inspection and food quality assessment. To capture images across a wide spectrum range, i.e. to achieve high spectral resolution, the sensor design sacrifices spatial resolution. With increasing mosaic size, this effect becomes increasingly detriment…
Features obtained from object recognition CNNs have been widely used for measuring perceptual similarities between images. Such differentiable metrics can be used as perceptual learning losses to train image enhancement models. However, the choice of the distance function between input and target features may have a consequential impact on the performance of the trained model. While using the norm of the difference between extracted features leads to limited hallucination of details, measuring the distance between distributions of features may generate more textures; yet also more unrealistic …
Enhancing Building Semantic Segmentation Accuracy with Super Resolution and Deep Learning: Investigating the Impact of Spatial Resolution on Various Datasets
The Hong Kong Polytechnic University（香港理工大学）；The University of Tokyo（东京大学）；Harbin Institute of Technology（哈尔滨工业大学）
Beihang University（北京航空航天大学）；Nanyang Technological University（南洋理工大学）；Institute for Infocomm Research（资讯通信研究院）
Spatially varying exposure (SVE) is a promising choice for high-dynamic-range (HDR) imaging (HDRI). The SVE-based HDRI, which is called single-shot HDRI, is an efficient solution to avoid ghosting artifacts. However, it is very challenging to restore a full-resolution HDR image from a real-world image with SVE because: a) only one-third of pixels with varying exposures are captured by camera in a Bayer pattern, b) some of the captured pixels are over- and under-exposed. For the former challenge, a spatially varying convolution (SVC) is designed to process the Bayer images carried with varying …
Is the center position fully capable of representing a pixel? There is nothing wrong to represent pixels with their centers in a discrete image representation, but it makes more sense to consider each pixel as the aggregation of signals from a local area in an image super-resolution (SR) context. Despite the great capability of coordinate-based implicit representation in the field of arbitrary-scale image SR, this area's nature of pixels is not fully considered. To this end, we propose integrated positional encoding (IPE), extending traditional positional encoding by aggregating frequency info…
University of Bern（伯尔尼大学）；University of Maryland, College Park（马里兰大学学院市分校）
In this paper we introduce a natural image prior that directly represents a Gaussian-smoothed version of the natural image distribution. We include our prior in a formulation of image restoration as a Bayes estimator that also allows us to solve noise-blind image restoration problems. We show that the gradient of our prior corresponds to the mean-shift vector on the natural image distribution. In addition, we learn the mean-shift vector field using denoising autoencoders, and use it in a gradient descent approach to perform Bayes risk minimization. We demonstrate competitive results for noise-…
Carnegie Mellon University（卡内基梅隆大学）；Indian Institute of Technology Roorkee（印度理工学院鲁尔基分校）；International Institute of Information Technology Bhubaneswar（布巴内斯瓦尔国际信息技术学院）
Single image super-resolution (SISR) is an ill-posed problem with an indeterminate number of valid solutions. Solving this problem with neural networks would require access to extensive experience, either presented as a large training set over natural images or a condensed representation from another pre-trained network. Perceptual loss functions, which belong to the latter category, have achieved breakthrough success in SISR and several other computer vision tasks. While perceptual loss plays a central role in the generation of photo-realistic images, it also produces undesired pattern artifa…
School of Minerals and Energy Resources Engineering, The University of New South Wales（新南威尔士大学矿物与能源资源工程学院）；CSIRO Energy（澳大利亚联邦科学与工业研究组织能源研究所）
X-ray micro-computed tomography (micro-CT) has been widely leveraged to characterise pore-scale geometry in subsurface porous rock. Recent developments in super resolution (SR) methods using deep learning allow the digital enhancement of low resolution (LR) images over large spatial scales, creating SR images comparable to the high resolution (HR) ground truth. This circumvents traditional resolution and field-of-view trade-offs. An outstanding issue is the use of paired (registered) LR and HR data, which is often required in the training step of such methods but is difficult to obtain. In thi…
Xperi Corporation（Xperi公司）；National University of Ireland, Galway（爱尔兰国立高威大学）
The process of obtaining high-resolution images from single or multiple low-resolution images of the same scene is of great interest for real-world image and signal processing applications. This study is about exploring the potential usage of deep learning based image super-resolution algorithms on thermal data for producing high quality thermal imaging results for in-cabin vehicular driver monitoring systems. In this work we have proposed and developed a novel multi-image super-resolution recurrent neural network to enhance the resolution and improve the quality of low-resolution thermal imag…
This work focuses on low bitrate video streaming scenarios (e.g. 50 - 200Kbps) where the video quality is severely compromised. We present a family of novel deep generative models for enhancing perceptual video quality of such streams by performing super-resolution while also removing compression artifacts. Our model, which we call SuperTran, consumes as input a single high-quality, high-resolution reference images in addition to the low-quality, low-resolution video stream. The model thus learns how to borrow or copy visual elements like textures from the reference image and fill in the remai…
University of Southern California（南加州大学）；Baidu Research (USA)（百度研究院（美国））
Normalizing flow models have been used successfully for generative image super-resolution (SR) by approximating complex distribution of natural images to simple tractable distribution in latent space through Invertible Neural Networks (INN). These models can generate multiple realistic SR images from one low-resolution (LR) input using randomly sampled points in the latent space, simulating the ill-posed nature of image upscaling where multiple high-resolution (HR) images correspond to the same LR. Lately, the invertible process in INN has also been used successfully by bidirectional image res…
Osaka Electro-Communication University（大阪电气通信大学）；Ritsumeikan University（立命馆大学）；Shiga University of Medical Science（滋贺医科大学）
Deep learning techniques have led to state-of-the-art image super resolution with natural images. Normally, pairs of high-resolution and low-resolution images are used to train the deep learning models. These techniques have also been applied to medical image super-resolution. The characteristics of medical images differ significantly from natural images in several ways. First, it is difficult to obtain high-resolution images for training in real clinical applications due to the limitations of imaging systems and clinical requirements. Second, other modal high-resolution images are available (…
The quality of images captured by smartphones is an important specification since smartphones are becoming ubiquitous as primary capturing devices. The traditional image signal processing (ISP) pipeline in a smartphone camera consists of several image processing steps performed sequentially to reconstruct a high quality sRGB image from the raw sensor data. These steps consist of demosaicing, denoising, white balancing, gamma correction, colour enhancement, etc. Since each of them are performed sequentially using hand-crafted algorithms, the residual error from each processing module accumulate…
Detail-Enhancing Framework for Reference-Based Image Super-Resolution
Shanghai Institution of Micro-system and Information Technology, University of Chinese Academy of Science（中国科学院大学上海微系统与信息技术研究所）；School of Information Science and Technology, Shanghai Tech University（上海科技大学信息科学与技术学院）；Computer Vision Laboratory, Department of Electrical Engineering, LiU, Sweden（瑞典林雪平大学电气工程系计算机视觉实验室）
ETH Zürich（苏黎世联邦理工学院）；KU Leuven（鲁汶大学）；Harbin Institute of Technology（哈尔滨工业大学）；The Hong Kong Polytechnic University（香港理工大学）
Recent works on plug-and-play image restoration have shown that a denoiser can implicitly serve as the image prior for model-based methods to solve many inverse problems. Such a property induces considerable advantages for plug-and-play image restoration (e.g., integrating the flexibility of model-based method and effectiveness of learning-based methods) when the denoiser is discriminatively learned via deep convolutional neural network (CNN) with large modeling capacity. However, while deeper and larger CNN models are rapidly gaining popularity, existing plug-and-play image restoration hinder…
Single Image Super Resolution (SISR) techniques based on Super Resolution Convolutional Neural Networks (SRCNN) are applied to micro-computed tomography (μCT) images of sandstone and carbonate rocks. Digital rock imaging is limited by the capability of the scanning device resulting in trade-offs between resolution and field of view, and super resolution methods tested in this study aim to compensate for these limits. SRCNN models SR-Resnet, Enhanced Deep SR (EDSR), and Wide-Activation Deep SR (WDSR) are used on the Digital Rock Super Resolution 1 (DRSRD1) Dataset of 4x downsampled images, comp…
Enhancing License Plate Super-Resolution: A Layout-Aware and Character-Driven Approach
Federal University of Paraná（巴拉那联邦大学）；Pontifical Catholic University of Paraná（巴拉那天主教大学）；Brazilian Federal Police（巴西联邦警察）；Federal University of Minas Gerais（米纳斯吉拉斯联邦大学）
As the popularity of mobile photography is growing constantly, lots of efforts are being invested now into building complex hand-crafted camera ISP solutions. In this work, we demonstrate that even the most sophisticated ISP pipelines can be replaced with a single end-to-end deep learning model trained without any prior knowledge about the sensor and optics used in a particular device. For this, we present PyNET, a novel pyramidal CNN architecture designed for fine-grained image restoration that implicitly learns to perform all ISP steps such as image demosaicing, denoising, white balancing, c…
In this paper, we propose a residual non-local attention network for high-quality image restoration. Without considering the uneven distribution of information in the corrupted images, previous methods are restricted by local convolutional operation and equal treatment of spatial- and channel-wise features. To address this issue, we design local and non-local attention blocks to extract features that capture the long-range dependencies between pixels and pay more attention to the challenging parts. Specifically, we design trunk branch and (non-)local mask branch in each (non-)local attention b…
BurstGP: Enhancing Raw Burst Image Super Resolution with Generative Priors
AI Center - Toronto, Samsung Electronics（多伦多AI中心、三星电子）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University Health Network（大学健康网络）；York University（约克大学）
UIUC（伊利诺伊大学厄巴纳-香槟分校）；Northeastern University（东北大学）；Google Brain（谷歌大脑）；ByteDance AI Lab（字节跳动人工智能实验室）；University of Oregon（俄勒冈大学）
Self-similarity refers to the image prior widely used in image restoration algorithms that small but similar patterns tend to occur at different locations and scales. However, recent advanced deep convolutional neural network based methods for image restoration do not take full advantage of self-similarities by relying on self-attention neural modules that only process information at the same scale. To solve this problem, we present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid. Inspired by the fact …
NeuroGAN-3D: Enhancing Intrinsic Functional Brain Networks via High-Fidelity 3D Generative Super-Resolution
Georgia State University（佐治亚州立大学）；Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS)（神经影像与数据科学转化研究三机构中心（TReNDS））；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）
The sensor to shooter timeline is affected by two main variables: satellite positioning and asset positioning. Speeding up satellite positioning by adding more sensors or by decreasing processing time is important only if there is a prepared shooter, otherwise the main source of time is getting the shooter into position. However, the intelligence community should work towards the exploitation of sensors to the highest speed and effectiveness possible. Achieving a high effectiveness while keeping speed high is a tradeoff that must be considered in the sensor to shooter timeline. In this paper w…
Division-of-focal-plane (DoFP) polarization imaging technical recently has been applied in many fields. However, the images captured by such sensors cannot be used directly because they suffer from instantaneous field-of-view errors and low resolution problem. This paper builds a fast DoFP demosaicing system with proposed progressive polarization demosaicing convolutional neural network (PPDN), which is specifically designed for edge-side GPU devices like Navidia Jetson TX2. The proposed network consists of two parts: reconstruction stage and refining stage. The former recovers four polarizati…
Enhancing Frequency for Single Image Super-Resolution with Learnable Separable Kernels
Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science；Technology, Shanghai, 200237, China. Department of Computer Science；Engineering, East China University of Science
Semantic-Guided Global-Local Collaborative Networks for Lightweight Image Super-Resolution
National and Local Joint Engineering Laboratory of Computer Aided Design（国家地方联合计算机辅助设计工程实验室）；School of Software Engineering, Dalian University（大连大学软件学院）；Centre for Advances in Reliability and Safety (CAiRS)（可靠性与安全先进中心）；School of Cyber Science and Technology, Sun Yat-Sen University（中山大学网络空间安全学院）
