HDR+ is an image processing pipeline presented by Google in 2016. At its core lies a denoising algorithm that uses a burst of raw images to produce a single higher quality image. Since it is designed as a versatile solution for smartphone cameras, it does not necessarily aim for the maximization of standard denoising metrics, but rather for the production of natural, visually pleasing images. In this article, we specifically discuss and analyze the HDR+ burst denoising algorithm architecture and the impact of its various parameters. With this publication, we provide an open source Python implementation of the algorithm, along with an interactive demo.
An image processing unit (IPU), or image signal processor (ISP) for high dynamic range (HDR) imaging usually consists of demosaicing, white balancing, lens shading correction, color correction, denoising, and tone-mapping. Besides noise from the imaging sensors, almost every step in the ISP introduces or amplifies noise in different ways, and denoising operators are designed to reduce the noise from these sources. Designed for dynamic range compressing, tone-mapping operators in an ISP can significantly amplify the noise level, especially for images captured in low-light conditions, making den…
University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Most consumer-grade digital cameras can only capture a limited range of luminance in real-world scenes due to sensor constraints. Besides, noise and quantization errors are often introduced in the imaging process. In order to obtain high dynamic range (HDR) images with excellent visual quality, the most common solution is to combine multiple images with different exposures. However, it is not always feasible to obtain multiple images of the same scene and most HDR reconstruction methods ignore the noise and quantization loss. In this work, we propose a novel learning-based approach using a spa…
We propose a patch-based method for the simultaneous denoising and fusion of a sequence of RAW multi-exposed images. A spatio-temporal criterion is used to select similar patches along the sequence, and a weighted principal component analysis permits to both denoise and fuse the multi exposed data. The overall strategy permits to denoise and fuse the set of images without the need of recovering each denoised image in the multi-exposure set, leading to a very efficient procedure. Several experiments show that the proposed method permits to obtain state-of-the-art fusion results with real RAW da…
Imperial College London（帝国理工学院）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Ghosting artifacts, motion blur, and low fidelity in highlight are the main challenges in High Dynamic Range (HDR) imaging from multiple Low Dynamic Range (LDR) images. These issues come from using the medium-exposed image as the reference frame in previous methods. To deal with them, we propose to use the under-exposed image as the reference to avoid these issues. However, the heavy noise in dark regions of the under-exposed image becomes a new problem. Therefore, we propose a joint HDR and denoising pipeline, containing two sub-networks: (i) a pre-denoising network (PreDNNet) to adaptively d…
MPI Informatik（马克斯·普朗克信息学研究所）；University College London（伦敦大学学院）
We seek to reconstruct sharp and noise-free high-dynamic range (HDR) video from a dual-exposure sensor that records different low-dynamic range (LDR) information in different pixel columns: Odd columns provide low-exposure, sharp, but noisy information; even columns complement this with less noisy, high-exposure, but motion-blurred data. Previous LDR work learns to deblur and denoise (DISTORTED->CLEAN) supervised by pairs of CLEAN and DISTORTED images. Regrettably, capturing DISTORTED sensor readings is time-consuming; as well, there is a lack of CLEAN HDR videos. We suggest a method to overco…
Duke University（杜克大学）；Télécom ParisTech, Université Paris-Saclay（巴黎高科电信学院，巴黎-萨克雷大学）；Université Paris Descartes（巴黎笛卡尔大学）；Universidad de la República（共和国大学）
Recently, impressive denoising results have been achieved by Bayesian approaches which assume Gaussian models for the image patches. This improvement in performance can be attributed to the use of per-patch models. Unfortunately such an approach is particularly unstable for most inverse problems beyond denoising. In this work, we propose the use of a hyperprior to model image patches, in order to stabilize the estimation procedure. There are two main advantages to the proposed restoration scheme: Firstly it is adapted to diagonal degradation matrices, and in particular to missing data problems…
National Center for Data Analytics and Artificial Intelligence（国家数据分析与人工智能中心）；KACST（阿卜杜勒-阿齐兹国王科技城）；Visual Computing Center（视觉计算中心）；KAUST（阿卜杜拉国王科技大学）
This paper study the reconstruction of High Dynamic Range (HDR) video from snapshot-coded LDR video. Constructing an HDR video requires restoring the HDR values for each frame and maintaining the consistency between successive frames. HDR image acquisition from single image capture, also known as snapshot HDR imaging, can be achieved in several ways. For example, the reconfigurable snapshot HDR camera is realized by introducing an optical element into the optical stack of the camera; by placing a coded mask at a small standoff distance in front of the sensor. High-quality HDR image can be reco…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
With the growing popularity of smartphones, capturing high-quality images is of vital importance to smartphones. The cameras of smartphones have small apertures and small sensor cells, which lead to the noisy images in low light environment. Denoising based on a burst of multiple frames generally outperforms single frame denoising but with the larger compututional cost. In this paper, we propose an efficient yet effective burst denoising system. We adopt a three-stage design: noise prior integration, multi-frame alignment and multi-frame denoising. First, we integrate noise prior by pre-proces…
Southeast University（东南大学）；Jiangsu Industrial Technology Research Institute（江苏集萃工业技术研究院）
Deep learning based image denoising methods have been extensively investigated. In this paper, attention mechanism enhanced kernel prediction networks (AME-KPNs) are proposed for burst image denoising, in which, nearly cost-free attention modules are adopted to first refine the feature maps and to further make a full use of the inter-frame and intra-frame redundancies within the whole image burst. The proposed AME-KPNs output per-pixel spatially-adaptive kernels, residual maps and corresponding weight maps, in which, the predicted kernels roughly restore clean pixels at their corresponding loc…
Demosaicking and denoising are the first steps of any camera image processing pipeline and are key for obtaining high quality RGB images. A promising current research trend aims at solving these two problems jointly using convolutional neural networks. Due to the unavailability of ground truth data these networks cannot be currently trained using real RAW images. Instead, they resort to simulated data. In this paper we present a method to learn demosaicking directly from mosaicked images, without requiring ground truth RGB data. We apply this to learn joint demosaicking and denoising only from…
Hong Kong Polytechnic University（香港理工大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）
Denoising and demosaicking are essential yet correlated steps to reconstruct a full color image from the raw color filter array (CFA) data. By learning a deep convolutional neural network (CNN), significant progress has been achieved to perform denoising and demosaicking jointly. However, most existing CNN-based joint denoising and demosaicking (JDD) methods work on a single image while assuming additive white Gaussian noise, which limits their performance on real-world applications. In this work, we study the JDD problem for real-world burst images, namely JDD-B. Considering the fact that the…
PhysHDR: When Lighting Meets Materials and Scene Geometry in HDR Reconstruction
School of Information Technology, Monash University, Malaysia（信息技术学院，墨尔本大学，马来西亚）；Faculty of Information Technology, Monash University, Australia（信息技术学院，墨尔本大学，澳大利亚）；Robotics and Autonomous Systems Lab, TCS Research, India（机器人与自主系统实验室，TCS研究，印度）
Computer Vision Lab, ETH Zurich（苏黎世联邦理工学院计算机视觉实验室）
We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image, which transforms the MAP objective to a deep feature space. The deep reparametrization allows us to directly model the image formation process in the latent space, and to integrate learned image priors into the prediction. Our approach thereby leverages the advantages of deep learning, while also benefiting from the principled multi-frame fusion provide…
The Hong Kong Polytechnic University（香港理工大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
Denoising and demosaicking are two essential steps to reconstruct a clean full-color image from the raw data. Recently, joint denoising and demosaicking (JDD) for burst images, namely JDD-B, has attracted much attention by using multiple raw images captured in a short time to reconstruct a single high-quality image. One key challenge of JDD-B lies in the robust alignment of image frames. State-of-the-art alignment methods in feature domain cannot effectively utilize the temporal information of burst images, where large shifts commonly exist due to camera and object motion. In addition, the hig…
Image resolution is an important criterion for many applications based on satellite imagery. In this work, we adapt a state-of-the-art kernel regression technique for smartphone camera burst super-resolution to satellites. This technique leverages the local structure of the image to optimally steer the fusion kernels, limiting blur in the final high-resolution prediction, denoising the image, and recovering details up to a zoom factor of 2. We extend this approach to the multi-exposure case to predict from a sequence of multi-exposure low-resolution frames a high-resolution and noise-free one.…
Technion - Israel Institute of Technology（以色列理工学院）
Mechanical image stabilization using actuated gimbals enables capturing long-exposure shots without suffering from blur due to camera motion. These devices, however, are often physically cumbersome and expensive, limiting their widespread use. In this work, we propose to digitally emulate a mechanically stabilized system from the input of a fast unstabilized camera. To exploit the trade-off between motion blur at long exposures and low SNR at short exposures, we train a CNN that estimates a sharp high-SNR image by aggregating a burst of noisy short-exposure frames, related by unknown motion. W…
Merging short-exposure frames can provide an image with reduced noise in low light conditions. However, how best to align images before merging is an open problem. To improve the performance of alignment, we propose an inertia-sensor aided algorithm for smartphone burst photography, which takes rotation and out-plane relative movement into account. To calculate homography between frames, a three by three rotation matrix is calculated from gyro data recorded by smartphone inertia sensor and three-dimensional translation vector are estimated by matched feature points detected from two frames. Th…
