High dynamic range (HDR) video reconstruction from sequences captured with alternating exposures is a very challenging problem. Existing methods often align low dynamic range (LDR) input sequence in the image space using optical flow, and then merge the aligned images to produce HDR output. However, accurate alignment and fusion in the image space are difficult due to the missing details in the over-exposed regions and noise in the under-exposed regions, resulting in unpleasing ghosting artifacts. To enable more accurate alignment and HDR fusion, we introduce a coarse-to-fine deep learning framework for HDR video reconstruction. Firstly, we perform coarse alignment and pixel blending in the image space to estimate the coarse HDR video. Secondly, we conduct more sophisticated alignment and temporal fusion in the feature space of the coarse HDR video to produce better reconstruction. Considering the fact that there is no publicly available dataset for quantitative and comprehensive evaluation of HDR video reconstruction methods, we collect such a benchmark dataset, which contains $97$ sequences of static scenes and 184 testing pairs of dynamic scenes. Extensive experiments show that our method outperforms previous state-of-the-art methods. Our dataset, code and model will be made publicly available.
University of Alberta（阿尔伯塔大学）；NUDT（国防科技大学）；Huawei Fields Lab（华为Fields实验室）
The recent development of the on-chip micro-polarizer technology has made it possible to acquire four spatially aligned and temporally synchronized polarization images with the same ease of operation as a conventional camera. In this paper, we investigate the use of this sensor technology in high-dynamic-range (HDR) imaging. Specifically, observing that natural light can be attenuated differently by varying the orientation of the polarization filter, we treat the multiple images captured by the polarization camera as a set captured under different exposure times. In our approach, we first stud…
Efficient HDR Reconstruction from Real-World Raw Images
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Shenzhen Bit Microelectronics Technology Co., Ltd（深圳比特微电子技术有限公司）
Recovering a high dynamic range (HDR) image from a single low dynamic range (LDR) input image is challenging due to missing details in under-/over-exposed regions caused by quantization and saturation of camera sensors. In contrast to existing learning-based methods, our core idea is to incorporate the domain knowledge of the LDR image formation pipeline into our model. We model the HDRto-LDR image formation pipeline as the (1) dynamic range clipping, (2) non-linear mapping from a camera response function, and (3) quantization. We then propose to learn three specialized CNNs to reverse these s…
High Dynamic Range Video Reconstruction from Single-Exposure Raw Sequences
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang Provincial Key Laboratory of Low Altitude Ubiquitous Networking Technology, Hangzhou Dianzi University（浙江省低空泛在组网技术重点实验室，杭州电子科技大学）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Institute of Technology（北京理工大学）
High Dynamic Range Image Reconstruction via Deep Explicit Polynomial Curve Estimation
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；SmartMore Technology（思谋科技）；Northwestern Polytechnical University（西北工业大学）；Zhejiang Lab（之江实验室）；Zhejiang University（浙江大学）
City University of Hong Kong（香港城市大学）；Dalian University of Technology（大连理工大学）；Tencent AI Lab（腾讯AI实验室）
Image correction aims to adjust an input image into a visually pleasing one. Existing approaches are proposed mainly from the perspective of image pixel manipulation. They are not effective to recover the details in the under/over exposed regions. In this paper, we revisit the image formation procedure and notice that the missing details in these regions exist in the corresponding high dynamic range (HDR) data. These details are well perceived by the human eyes but diminished in the low dynamic range (LDR) domain because of the tone mapping process. Therefore, we formulate the image correction…
High Dynamic Range Imaging Based on an Asymmetric Event-SVE Camera System
College of Aerospace Science and Engineering（航空航天科学与工程学院）；National University of Defense Technology（国防科技大学）；Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation（湖南省图像测量与视觉导航重点实验室）
The University of Tokyo（东京大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织数据61研究所）；Australian National University（澳大利亚国立大学）
This paper tackles high-dynamic-range (HDR) image reconstruction given only a single low-dynamic-range (LDR) image as input. While the existing methods focus on minimizing the mean-squared-error (MSE) between the target and reconstructed images, we minimize a hybrid loss that consists of perceptual and adversarial losses in addition to HDR-reconstruction loss. The reconstruction loss instead of MSE is more suitable for HDR since it puts more weight on both over- and under- exposed areas. It makes the reconstruction faithful to the input. Perceptual loss enables the networks to utilize knowledg…
Reconstruction of high-quality HDR images is at the core of modern computational photography. Significant progress has been made with multi-frame HDR reconstruction methods, producing high-resolution, rich and accurate color reconstructions with high-frequency details. However, they are still prone to fail in dynamic or largely over-exposed scenes, where frame misalignment often results in visible ghosting artifacts. Recent approaches attempt to alleviate this by utilizing an event-based camera (EBC), which measures only binary changes of illuminations. Despite their desirable high temporal re…
This paper proposes a novel learning based high-dynamic-range (HDR) reconstruction method using a polarization camera. We utilize a previous observation that polarization filters with different orientations can attenuate natural light differently, and we treat the multiple images acquired by the polarization camera as a set acquired under different exposure times, to introduce the development of solutions for the HDR reconstruction problem. We propose a deep HDR reconstruction framework with a feature masking mechanism that uses polarimetric cues available from the polarization camera, called …
Boosting HDR Image Reconstruction via Semantic Knowledge Transfer
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院）；Shaanxi University of Science & Technology（陕西科技大学）；Xi’an University of Architecture and Technology（西安建筑科技大学）；Chinese University of Hong Kong（香港中文大学）
Single-Image HDR Reconstruction Assisted Ghost Suppression and Detail Preservation Network for Multi-Exposure HDR Imaging
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）
F2HDR: Two-Stage HDR Video Reconstruction via Flow Adapter and Physical Motion Modeling
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；State Key Laboratory of Smart Power Distribution Equipment and System, Tianjin University（天津大学智能电力分配设备与系统国家重点实验室）；Huawei Technologies Co., Ltd.（华为技术有限公司）
Bio-SFT: Asymmetric Cortical Guidance and Retinal Adaptation for Robust HDR Reconstruction
College of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）；School of Computer Science, Nankai University（南开大学计算机科学学院）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）
Australian National University（澳大利亚国立大学）；ETH Zurich（苏黎世联邦理工学院）；Monash University（莫纳什大学）
Event cameras are novel, bio-inspired visual sensors, whose pixels output asynchronous and independent timestamped spikes at local intensity changes, called 'events'. Event cameras offer advantages over conventional frame-based cameras in terms of latency, high dynamic range (HDR) and temporal resolution. Until recently, event cameras have been limited to outputting events in the intensity channel, however, recent advances have resulted in the development of color event cameras, such as the Color-DAVIS346. In this work, we present and release the first Color Event Camera Dataset (CED), contain…
Fuzhou University（福州大学）；City University of Hong Kong（香港城市大学）
Synthesizing high dynamic range (HDR) images from multiple low-dynamic range (LDR) exposures in dynamic scenes is challenging. There are two major problems caused by the large motions of foreground objects. One is the severe misalignment among the LDR images. The other is the missing content due to the over-/under-saturated regions caused by the moving objects, which may not be easily compensated for by the multiple LDR exposures. Thus, it requires the HDR generation model to be able to properly fuse the LDR images and restore the missing details without introducing artifacts. To address these…
Northwestern University（西北大学）；Wormpex AI Research（万沛克斯人工智能研究院）
HDR reconstruction is an important task in computer vision with many industrial needs. The traditional approaches merge multiple exposure shots to generate HDRs that correspond to the physical quantity of illuminance of the scene. However, the tedious capturing process makes such multi-shot approaches inconvenient in practice. In contrast, recent single-shot methods predict a visually appealing HDR from a single LDR image through deep learning. But it is not clear whether the previously mentioned physical properties would still hold, without training the network to explicitly model them. In th…
University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Lab（上海人工智能实验室）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
Most consumer-grade digital cameras can only capture a limited range of luminance in real-world scenes due to sensor constraints. Besides, noise and quantization errors are often introduced in the imaging process. In order to obtain high dynamic range (HDR) images with excellent visual quality, the most common solution is to combine multiple images with different exposures. However, it is not always feasible to obtain multiple images of the same scene and most HDR reconstruction methods ignore the noise and quantization loss. In this work, we propose a novel learning-based approach using a spa…
Indian Institute of Technology（印度理工学院）；University of Colorado（科罗拉多大学）
High dynamic range (HDR) videos provide a more visually realistic experience than the standard low dynamic range (LDR) videos. Despite having significant progress in HDR imaging, it is still a challenging task to capture high-quality HDR video with a conventional off-the-shelf camera. Existing approaches rely entirely on using dense optical flow between the neighboring LDR sequences to reconstruct an HDR frame. However, they lead to inconsistencies in color and exposure over time when applied to alternating exposures with noisy frames. In this paper, we propose an end-to-end GAN-based framewor…
Indian Institute of Technology Gandhinagar（甘地讷格尔印度理工学院）
High dynamic range (HDR) image generation from a single exposure low dynamic range (LDR) image has been made possible due to the recent advances in Deep Learning. Various feed-forward Convolutional Neural Networks (CNNs) have been proposed for learning LDR to HDR representations. To better utilize the power of CNNs, we exploit the idea of feedback, where the initial low level features are guided by the high level features using a hidden state of a Recurrent Neural Network. Unlike a single forward pass in a conventional feed-forward network, the reconstruction from LDR to HDR in a feedback netw…
School of Computer Science, Wuhan University（武汉大学计算机学院）；JD Finance America Corporation（京东金融美国公司）；SRI International（斯坦福国际研究院）
It is very challenging to reconstruct a high dynamic range (HDR) from a low dynamic range (LDR) image as an ill-posed problem. This paper proposes a luminance attentive network named LANet for HDR reconstruction from a single LDR image. Our method is based on two fundamental observations: (1) HDR images stored in relative luminance are scale-invariant, which means the HDR images will hold the same information when multiplied by any positive real number. Based on this observation, we propose a novel normalization method called " HDR calibration " for HDR images stored in relative luminance, cal…
We propose a novel high dynamic range (HDR) video reconstruction method with new tri-exposure quad-bayer sensors. Thanks to the larger number of exposure sets and their spatially uniform deployment over a frame, they are more robust to noise and spatial artifacts than previous spatially varying exposure (SVE) HDR video methods. Nonetheless, the motion blur from longer exposures, the noise from short exposures, and inherent spatial artifacts of the SVE methods remain huge obstacles. Additionally, temporal coherence must be taken into account for the stability of video reconstruction. To tackle …
VinAI Research（VinAI研究院）；University of California San Diego（加州大学圣地亚哥分校）
High dynamic range (HDR) imaging is an indispensable technique in modern photography. Traditional methods focus on HDR reconstruction from multiple images, solving the core problems of image alignment, fusion, and tone mapping, yet having a perfect solution due to ghosting and other visual artifacts in the reconstruction. Recent attempts at single-image HDR reconstruction show a promising alternative: by learning to map pixel values to their irradiance using a neural network, one can bypass the align-and-merge pipeline completely yet still obtain a high-quality HDR image. In this work, we prop…
High dynamic range (HDR) imaging is one of the biggest achievements in modern photography. Traditional solutions to HDR imaging are designed for and applied to CMOS image sensors (CIS). However, the mainstream one-micron CIS cameras today generally have a high read noise and low frame-rate. These, in turn, limit the acquisition speed and quality, making the cameras slow in the HDR mode. In this paper, we propose a new computational photography technique for HDR imaging. Recognizing the limitations of CIS, we use the Quanta Image Sensor (QIS) to trade the spatial-temporal resolution with bit-de…
National Center for Data Analytics and Artificial Intelligence（国家数据分析与人工智能中心）；KACST（阿卜杜勒-阿齐兹国王科技城）；Visual Computing Center（视觉计算中心）；KAUST（阿卜杜拉国王科技大学）
This paper study the reconstruction of High Dynamic Range (HDR) video from snapshot-coded LDR video. Constructing an HDR video requires restoring the HDR values for each frame and maintaining the consistency between successive frames. HDR image acquisition from single image capture, also known as snapshot HDR imaging, can be achieved in several ways. For example, the reconfigurable snapshot HDR camera is realized by introducing an optical element into the optical stack of the camera; by placing a coded mask at a small standoff distance in front of the sensor. High-quality HDR image can be reco…
Mapping a single exposure low dynamic range (LDR) image into a high dynamic range (HDR) is considered among the most strenuous image to image translation tasks due to exposure-related missing information. This study tackles the challenges of single-shot LDR to HDR mapping by proposing a novel two-stage deep network. Notably, our proposed method aims to reconstruct an HDR image without knowing hardware information, including camera response function (CRF) and exposure settings. Therefore, we aim to perform image enhancement task like denoising, exposure correction, etc., in the first stage. Add…
Communication University of China（中国传媒大学）；Peng Cheng Laboratory（鹏城实验室）
High dynamic range (HDR) image is widely-used in graphics and photography due to the rich information it contains. Recently the community has started using deep neural network (DNN) to reconstruct standard dynamic range (SDR) images into HDR. Albeit the superiority of current DNN-based methods, their application scenario is still limited: (1) heavy model impedes real-time processing, and (2) inapplicable to legacy SDR content with more degradation types. Therefore, we propose a lightweight DNN-based method trained to tackle legacy SDR. For better design, we reform the problem modeling and emph…
HDRAgent: An Agentic Framework for Multi-Exposure HDR Imaging
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Shenzhen Research Institute, Northwestern Polytechnical University（西北工业大学深圳研究院）；Zhejiang University（浙江大学）；Camera Group, DJI（大疆相机部门）
IIIT Hyderabad（海得拉巴国际信息技术学院）；University of California, Los Angeles（加州大学洛杉矶分校）；Georgia Tech（佐治亚理工学院）；IIT Gandhinagar（甘地纳格尔印度理工学院）
Due to hardware constraints, standard off-the-shelf digital cameras suffers from low dynamic range (LDR) and low frame per second (FPS) outputs. Previous works in high dynamic range (HDR) video reconstruction uses sequence of alternating exposure LDR frames as input, and align the neighbouring frames using optical flow based networks. However, these methods often result in motion artifacts in challenging situations. This is because, the alternate exposure frames have to be exposure matched in order to apply alignment using optical flow. Hence, over-saturation and noise in the LDR frames result…
Linköping University（林雪平大学）；University of Cambridge（剑桥大学）
Single-image high dynamic range (SI-HDR) reconstruction has recently emerged as a problem well-suited for deep learning methods. Each successive technique demonstrates an improvement over existing methods by reporting higher image quality scores. This paper, however, highlights that such improvements in objective metrics do not necessarily translate to visually superior images. The first problem is the use of disparate evaluation conditions in terms of data and metric parameters, calling for a standardized protocol to make it possible to compare between papers. The second problem, which forms …
Wavelet-Domain Masked Image Modeling for Color-Consistent HDR Video Reconstruction
School of Computer Science and Technology and the Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University（计算机科学与技术学院和嵌入式系统与服务计算重点实验室，教育部，同济大学）；College of Electronic and Information Engineering, the School of Computer Science and Technology, and the Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University（电子与信息工程学院、计算机科学与技术学院和嵌入式系统与服务计算重点实验室，教育部，同济大学）；Pengcheng Laboratory（鹏城实验室）
Capturing scenes with a high dynamic range is crucial to reproducing images that appear similar to those seen by the human visual system. Despite progress in developing data-driven deep learning approaches for converting low dynamic range images to high dynamic range images, existing approaches are limited by the assumption that all conversions are governed by the same nonlinear mapping. To address this problem, we propose "Model-Agnostic Meta-Learning for HDR Image Reconstruction" (MetaHDR), which applies meta-learning to the LDR-to-HDR conversion problem using existing HDR datasets. Our key …
A Cycle Ride to HDR: Semantics Aware Self-Supervised Framework for Unpaired LDR-to-HDR Image Reconstruction
School of Information Technology, Monash University Malaysia（莫纳什大学马来西亚分校信息科技学院）；Faculty of Information Technology, Monash University（莫纳什大学信息科技学院）；Robotics and Autonomous Systems Lab, TCS Research（TCS研究机器人与自主系统实验室）
University of Science and Technology of China（中国科学技术大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Existing deep learning based HDRTV reconstruction methods assume one kind of tone mapping operators (TMOs) as the degradation procedure to synthesize SDRTV-HDRTV pairs for supervised training. In this paper, we argue that, although traditional TMOs exploit efficient dynamic range compression priors, they have several drawbacks on modeling the realistic degradation: information over-preservation, color bias and possible artifacts, making the trained reconstruction networks hard to generalize well to real-world cases. To solve this problem, we propose a learning-based data synthesis approach to …
EventHDR: from Event to High-Speed HDR Videos and Beyond
Beijing Institute of Technology（北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；University of Tsukuba（筑波大学）；Next Generation Artificial Intelligence Research Center, the University of Tokyo（东京大学下一代人工智能研究中心）
