Learning RAW-to-sRGB mapping has drawn increasing attention in recent years, wherein an input raw image is trained to imitate the target sRGB image captured by another camera. However, the severe color inconsistency makes it very challenging to generate well-aligned training pairs of input raw and target sRGB images. While learning with inaccurately aligned supervision is prone to causing pixel shift and producing blurry results. In this paper, we circumvent such issue by presenting a joint learning model for image alignment and RAW-to-sRGB mapping. To diminish the effect of color inconsistency in image alignment, we introduce to use a global color mapping (GCM) module to generate an initial sRGB image given the input raw image, which can keep the spatial location of the pixels unchanged, and the target sRGB image is utilized to guide GCM for converting the color towards it. Then a pre-trained optical flow estimation network (e.g., PWC-Net) is deployed to warp the target sRGB image to align with the GCM output. To alleviate the effect of inaccurately aligned supervision, the warped target sRGB image is leveraged to learn RAW-to-sRGB mapping. When training is done, the GCM module and optical flow network can be detached, thereby bringing no extra computation cost for inference. Experiments show that our method performs favorably against state-of-the-arts on ZRR and SR-RAW datasets. With our joint learning model, a light-weight backbone can achieve better quantitative and qualitative performance on ZRR dataset. Codes are available at https://github.com/cszhilu1998/RAW-to-sRGB.
ISPDiffuser: Learning RAW-to-sRGB Mappings with Texture-Aware Diffusion Models and Histogram-Guided Color Consistency
School of Aeronautics and Astronautics, Sichuan University（四川大学航空航天学院）；University of Electronic Science and Technology of China（电子科技大学）；Megvii Technology（旷视科技）
York University（约克大学）；Samsung AI Center – Toronto（三星人工智能中心多伦多分部）
Most camera images are rendered and saved in the standard RGB (sRGB) format by the camera's hardware. Due to the in-camera photo-finishing routines, nonlinear sRGB images are undesirable for computer vision tasks that assume a direct relationship between pixel values and scene radiance. For such applications, linear raw-RGB sensor images are preferred. Saving images in their raw-RGB format is still uncommon due to the large storage requirement and lack of support by many imaging applications. Several "raw reconstruction" methods have been proposed that utilize specialized metadata sampled from…
The Hong Kong University of Science and Technology（香港科技大学）
Unprocessed RAW data is a highly valuable image format for image editing and computer vision. However, since the file size of RAW data is huge, most users can only get access to processed and compressed sRGB images. To bridge this gap, we design an Invertible Image Signal Processing (InvISP) pipeline, which not only enables rendering visually appealing sRGB images but also allows recovering nearly perfect RAW data. Due to our framework's inherent reversibility, we can reconstruct realistic RAW data instead of synthesizing RAW data from sRGB images without any memory overhead. We also integrate…
York University（约克大学）；Vector Institute（向量研究所）；Samsung AI Center–Toronto（三星人工智能中心（多伦多））
Noise modeling and reduction are fundamental tasks in low-level computer vision. They are particularly important for smartphone cameras relying on small sensors that exhibit visually noticeable noise. There has recently been renewed interest in using data-driven approaches to improve camera noise models via neural networks. These data-driven approaches target noise present in the raw-sensor image before it has been processed by the camera's image signal processor (ISP). Modeling noise in the RAW-rgb domain is useful for improving and testing the in-camera denoising algorithm; however, there ar…
Color Matching Using Hypernetwork-Based Kolmogorov-Arnold Networks
Samara National Research University（萨马拉国立研究大学）；University of Würzburg（维尔茨堡大学）；York University（约克大学）；Institute for Information Transmission Problems RAS（俄罗斯科学院信息传输问题研究所）；Moscow Institute of Physics and Technologies（莫斯科物理技术学院）；Artificial Intelligence Research Institute（人工智能研究院）
本文提出 cmKAN 框架，利用超网络生成空间权重图控制 KAN 非线性样条，完成多种相机色彩空间下的颜色匹配，并构建大规模成对图像数据集，性能平均提升 37.3%。
Visual Autoregressive Priors for RAW-to-sRGB Image Signal Processing
OmniVision-IDT Joint Laboratory for Intelligent Image Sensing（豪威科技-东方理工智能图像感知联合实验室）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波市空间智能与数字衍生重点实验室）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波东方理工数字孪生研究院）；Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin（浙江省工业智能与数字孪生重点实验室）；The Hong Kong Polytechnic University（香港理工大学）
We present a novel deep learning framework that models the scene dependent image processing inside cameras. Often called as the radiometric calibration, the process of recovering RAW images from processed images (JPEG format in the sRGB color space) is essential for many computer vision tasks that rely on physically accurate radiance values. All previous works rely on the deterministic imaging model where the color transformation stays the same regardless of the scene and thus they can only be applied for images taken under the manual mode. In this paper, we propose a data-driven approach to l…
Nanyang Technological University（南洋理工大学）；Peng Cheng Lab（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）
While raw images exhibit advantages over sRGB images (e.g., linearity and fine-grained quantization level), they are not widely used by common users due to the large storage requirements. Very recent works propose to compress raw images by designing the sampling masks in the raw image pixel space, leading to suboptimal image representations and redundant metadata. In this paper, we propose a novel framework to learn a compact representation in the latent space serving as the metadata in an end-to-end manner. Furthermore, we propose a novel sRGB-guided context model with improved entropy estima…
In recent years, real image super-resolution (SR) has achieved promising results due to the development of SR datasets and corresponding real SR methods. In contrast, the field of real video SR is lagging behind, especially for real raw videos. Considering the superiority of raw image SR over sRGB image SR, we construct a real-world raw video SR (Real-RawVSR) dataset and propose a corresponding SR method. We utilize two DSLR cameras and a beam-splitter to simultaneously capture low-resolution (LR) and high-resolution (HR) raw videos with 2x, 3x, and 4x magnifications. There are 450 video pairs…
In this paper, we make the first benchmark effort to elaborate on the superiority of using RAW images in the low light enhancement and develop a novel alternative route to utilize RAW images in a more flexible and practical way. Inspired by a full consideration on the typical image processing pipeline, we are inspired to develop a new evaluation framework, Factorized Enhancement Model (FEM), which decomposes the properties of RAW images into measurable factors and provides a tool for exploring how properties of RAW images affect the enhancement performance empirically. The empirical benchmark …
Efficient RAW Image Deblurring with Adaptive Frequency Modulation
School of Mathematics, Tianjin University（天津大学数学学院）；School of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）
Deep learning-based blind image deblurring plays an essential role in solving image blur since all existing kernels are limited in modeling the real world blur. Thus far, researchers focus on powerful models to handle the deblurring problem and achieve decent results. For this work, in a new aspect, we discover the great opportunity for image enhancement (e.g., deblurring) directly from RAW images and investigate novel neural network structures benefiting RAW-based learning. However, to the best of our knowledge, there is no available RAW image deblurring dataset. Therefore, we built a new dat…
In order to get raw images of high quality for downstream Image Signal Process (ISP), in this paper we present an Efficient Locally Multiplicative Transformer called ELMformer for raw image restoration. ELMformer contains two core designs especially for raw images whose primitive attribute is single-channel. The first design is a Bi-directional Fusion Projection (BFP) module, where we consider both the color characteristics of raw images and spatial structure of single-channel. The second one is that we propose a Locally Multiplicative Self-Attention (L-MSA) scheme to effectively deliver infor…
BurstGP: Enhancing Raw Burst Image Super Resolution with Generative Priors
AI Center - Toronto, Samsung Electronics（多伦多AI中心、三星电子）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University Health Network（大学健康网络）；York University（约克大学）
In recent years, the supervised learning strategy for real noisy image denoising has been emerging and has achieved promising results. In contrast, realistic noise removal for raw noisy videos is rarely studied due to the lack of noisy-clean pairs for dynamic scenes. Clean video frames for dynamic scenes cannot be captured with a long-exposure shutter or averaging multi-shots as was done for static images. In this paper, we solve this problem by creating motions for controllable objects, such as toys, and capturing each static moment for multiple times to generate clean video frames. In this w…
RAWIC: Bit-Depth Adaptive Lossless Raw Image Compression
School of Electronics, Peking University（北京大学电子学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；University of Chinese Academy of Sciences（中国科学院大学）
RAW image datasets are more suitable than the standard RGB image datasets for the ill-posed inverse problems in low-level vision, but not common in the literature. There are also a few studies to focus on mapping sRGB images to RAW format. Mapping from sRGB to RAW format could be a relevant domain for reverse style transferring since the task is an ill-posed reversing problem. In this study, we seek an answer to the question: Can the ISP operations be modeled as the style factor in an end-to-end learning pipeline? To investigate this idea, we propose a novel architecture, namely RST-ISP-Net, f…
ETH Zürich（苏黎世联邦理工学院）；University of Würzburg（维尔茨堡大学）；KU Leuven（鲁汶大学）
We propose a trainable Image Signal Processing (ISP) framework that produces DSLR quality images given RAW images captured by a smartphone. To address the color misalignments between training image pairs, we employ a color-conditional ISP network and optimize a novel parametric color mapping between each input RAW and reference DSLR image. During inference, we predict the target color image by designing a color prediction network with efficient Global Context Transformer modules. The latter effectively leverage global information to learn consistent color and tone mappings. We further propose …
Cameras currently allow access to two image states: (i) a minimally processed linear raw-RGB image state (i.e., raw sensor data) or (ii) a highly-processed nonlinear image state (e.g., sRGB). There are many computer vision tasks that work best with a linear image state, such as image deblurring and image dehazing. Unfortunately, the vast majority of images are saved in the nonlinear image state. Because of this, a number of methods have been proposed to "unprocess" nonlinear images back to a raw-RGB state. However, existing unprocessing methods have a drawback because raw-RGB images are sensor…
Real-Time Scene-Adaptive Tone Mapping for High-Dynamic Range Object Detection
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；Tianmushan Laboratory, Beihang University（北京航空航天大学天目山实验室）；School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Point Spread Technology（点扩散科技）
Real-world low-light images suffer from two main degradations, namely, inevitable noise and poor visibility. Since the noise exhibits different levels, its estimation has been implemented in recent works when enhancing low-light images from raw Bayer space. When it comes to sRGB color space, the noise estimation becomes more complicated due to the effect of the image processing pipeline. Nevertheless, most existing enhancing algorithms in sRGB space only focus on the low visibility problem or suppress the noise under a hypothetical noise level, leading them impractical due to the lack of robus…
Compared with raw images, the more common JPEG images are less useful for machine vision algorithms and professional photographers because JPEG-sRGB does not preserve a linear relation between pixel values and the light measured from the scene. A camera is said to be radiometrically calibrated if there is a computational model which can predict how the raw linear sensor image is mapped to the corresponding rendered image (e.g. JPEGs) and vice versa. This paper begins with the observation that the rank order of pixel values are mostly preserved post colour correction. We show that this observat…
Image recognition models that work in challenging environments (e.g., extremely dark, blurry, or high dynamic range conditions) must be useful. However, creating training datasets for such environments is expensive and hard due to the difficulties of data collection and annotation. It is desirable if we could get a robust model without the need for hard-to-obtain datasets. One simple approach is to apply data augmentation such as color jitter and blur to standard RGB (sRGB) images in simple scenes. Unfortunately, this approach struggles to yield realistic images in terms of pixel intensity and…
MambaRaw: Selective State Space Modeling for Efficient 4K Raw Image Reconstruction
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Dalian University of Technology（大连理工大学）；Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science, Peking University（北京大学计算机科学学院）
School of Aeronautics and Astronautics, Sichuan University（四川大学航空宇航学院）；University of Electronic Science and Technology of China（电子科技大学）；Shanghai Jiao Tong University（上海交通大学）；National Innovation Center for UHD Video Technology（超高清视频技术国家创新中心）
Retinex-RAWMamba: Bridging Demosaicing and Denoising for Low-Light RAW Image Enhancement
Institute of Advanced Technology, University of Science and Technology of China（科学技术大学先进技术研究院）；School of Computer and Artificial Intelligence, Beijing Technology and Business University（北京科技与商务大学计算机与人工智能学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；University of Science and Technology of China（科学技术大学）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）
Learning Arbitrary-Scale RAW Image Downscaling with Wavelet-based Recurrent Reconstruction
School of Aeronautics and Astronautics, Sichuan University（四川大学航空宇航学院）；Xiaomi Inc.（小米公司）；University of Electronic Science and Technology of China（电子科技大学）
