While raw images exhibit advantages over sRGB images (e.g., linearity and fine-grained quantization level), they are not widely used by common users due to the large storage requirements. Very recent works propose to compress raw images by designing the sampling masks in the raw image pixel space, leading to suboptimal image representations and redundant metadata. In this paper, we propose a novel framework to learn a compact representation in the latent space serving as the metadata in an end-to-end manner. Furthermore, we propose a novel sRGB-guided context model with improved entropy estimation strategies, which leads to better reconstruction quality, smaller size of metadata, and faster speed. We illustrate how the proposed raw image compression scheme can adaptively allocate more bits to image regions that are important from a global perspective. The experimental results show that the proposed method can achieve superior raw image reconstruction results using a smaller size of the metadata on both uncompressed sRGB images and JPEG images.
Recent research on learning a mapping between raw Bayer images and RGB images has progressed with the development of deep convolutional neural networks. A challenging data set namely the Zurich Raw-to-RGB data set (ZRR) has been released in the AIM 2019 raw-to-RGB mapping challenge. In ZRR, input raw and target RGB images are captured by two different cameras and thus not perfectly aligned. Moreover, camera metadata such as white balance gains and color correction matrix are not provided, which makes the challenge more difficult. In this paper, we explore an effective network structure and a l…
Cameras capture sensor RAW images and transform them into pleasant RGB images, suitable for the human eyes, using their integrated Image Signal Processor (ISP). Numerous low-level vision tasks operate in the RAW domain (e.g. image denoising, white balance) due to its linear relationship with the scene irradiance, wide-range of information at 12bits, and sensor designs. Despite this, RAW image datasets are scarce and more expensive to collect than the already large and public RGB datasets. This paper introduces the AIM 2022 Challenge on Reversed Image Signal Processing and RAW Reconstruction. W…
Inria（法国国家信息与自动化研究所）；DI/ENS (ENS-PSL, CNRS, Inria)（巴黎高等师范学院计算机系（巴黎文理研究大学巴黎高师、法国国家科学研究中心、法国国家信息与自动化研究所））；Université Paris-Saclay（巴黎萨克雷大学）；ENS Paris-Saclay（巴黎-萨克雷高等师范学院）；Centre Borelli（博雷利中心）；Center for Data Science, New York University（纽约大学数据科学中心）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔理工学院）；LJK（让·昆茨曼实验室）
Photographs captured by smartphones and mid-range cameras have limited spatial resolution and dynamic range, with noisy response in underexposed regions and color artefacts in saturated areas. This paper introduces the first approach (to the best of our knowledge) to the reconstruction of high-resolution, high-dynamic range color images from raw photographic bursts captured by a handheld camera with exposure bracketing. This method uses a physically-accurate model of image formation to combine an iterative optimization algorithm for solving the corresponding inverse problem with a learned imag…
York University（约克大学）；Samsung AI Center – Toronto（三星人工智能中心多伦多分部）
Most camera images are rendered and saved in the standard RGB (sRGB) format by the camera's hardware. Due to the in-camera photo-finishing routines, nonlinear sRGB images are undesirable for computer vision tasks that assume a direct relationship between pixel values and scene radiance. For such applications, linear raw-RGB sensor images are preferred. Saving images in their raw-RGB format is still uncommon due to the large storage requirement and lack of support by many imaging applications. Several "raw reconstruction" methods have been proposed that utilize specialized metadata sampled from…
Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real data requires careful consideration of the noise properties of image sensors, the other aspects of a camera's image processing pipeline (gain, color correction, tone mapping, etc) are often overlooked, despite their significant eff…
RAWIC: Bit-Depth Adaptive Lossless Raw Image Compression
School of Electronics, Peking University（北京大学电子学院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Inria（法国国家信息与自动化研究所）；DIENS (ENS-PSL, CNRS, Inria)（DIENS（巴黎高等师范学院-PSL，法国国家科学研究中心，法国国家信息与自动化研究所））；Univ. Grenoble Alpes（格勒诺布尔-阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；LJK（LJK（让·库尔坦研究所））；Center for Data Science, New York University（纽约大学数据科学中心）
This presentation addresses the problem of reconstructing a high-resolution image from multiple lower-resolution snapshots captured from slightly different viewpoints in space and time. Key challenges for solving this problem include (i) aligning the input pictures with sub-pixel accuracy, (ii) handling raw (noisy) images for maximal faithfulness to native camera data, and (iii) designing/learning an image prior (regularizer) well suited to the task. We address these three challenges with a hybrid algorithm building on the insight from Wronski et al. that aliasing is an ally in this setting, w…
Beijing Institute of Technology（北京理工大学）；Microsoft Research（微软研究院）
Lacking rich and realistic data, learned single image denoising algorithms generalize poorly to real raw images that do not resemble the data used for training. Although the problem can be alleviated by the heteroscedastic Gaussian model for noise synthesis, the noise sources caused by digital camera electronics are still largely overlooked, despite their significant effect on raw measurement, especially under extremely low-light condition. To address this issue, we present a highly accurate noise formation model based on the characteristics of CMOS photosensors, thereby enabling us to synthes…
University of Malta（马耳他大学）；University of Edinburgh（爱丁堡大学）
Convolutional Neural Networks (CNNs) have achieved impressive results across many super-resolution (SR) and image restoration tasks. While many such networks can upscale low-resolution (LR) images using just the raw pixel-level information, the ill-posed nature of SR can make it difficult to accurately super-resolve an image which has undergone multiple different degradations. Additional information (metadata) describing the degradation process (such as the blur kernel applied, compression level, etc.) can guide networks to super-resolve LR images with higher fidelity to the original source. P…
Unprocessed sensor outputs (RAW images) potentially improve both low-level and high-level computer vision algorithms, but the lack of large-scale RAW image datasets is a barrier to research. Thus, reversed Image Signal Processing (ISP) which converts existing RGB images into RAW images has been studied. However, most existing methods require camera-specific metadata or paired RGB and RAW images to model the conversion, and they are not always available. In addition, there are issues in handling diverse ISPs and recovering global illumination. To tackle these limitations, we propose a self-supe…
MambaRaw: Selective State Space Modeling for Efficient 4K Raw Image Reconstruction
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Dalian University of Technology（大连理工大学）；Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science, Peking University（北京大学计算机科学学院）
Understanding scenes in movies is crucial for a variety of applications such as video moderation, search, and recommendation. However, labeling individual scenes is a time-consuming process. In contrast, movie level metadata (e.g., genre, synopsis, etc.) regularly gets produced as part of the film production process, and is therefore significantly more commonly available. In this work, we propose a novel contrastive learning approach that uses movie metadata to learn a general-purpose scene representation. Specifically, we use movie metadata to define a measure of movie similarity, and use it …
This paper reviews the second AIM learned ISP challenge and provides the description of the proposed solutions and results. The participating teams were solving a real-world RAW-to-RGB mapping problem, where to goal was to map the original low-quality RAW images captured by the Huawei P20 device to the same photos obtained with the Canon 5D DSLR camera. The considered task embraced a number of complex computer vision subtasks, such as image demosaicing, denoising, white balancing, color and contrast correction, demoireing, etc. The target metric used in this challenge combined fidelity scores …
SentiFormer: Metadata Enhanced Transformer for Image Sentiment Analysis
University of Science and Technology of China（中国科学技术大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining
Department of Biomedical Engineering, Southern University of Science and Technology, China（南方科技大学生物医学工程系，中国）；School of Biomedical Engineering, Shenzhen University, China（深圳大学生物医学工程学院，中国）
Visual Autoregressive Priors for RAW-to-sRGB Image Signal Processing
OmniVision-IDT Joint Laboratory for Intelligent Image Sensing（豪威科技-东方理工智能图像感知联合实验室）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波市空间智能与数字衍生重点实验室）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo（宁波东方理工数字孪生研究院）；Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin（浙江省工业智能与数字孪生重点实验室）；The Hong Kong Polytechnic University（香港理工大学）
Existing neural networks for computer vision tasks are vulnerable to adversarial attacks: adding imperceptible perturbations to the input images can fool these methods to make a false prediction on an image that was correctly predicted without the perturbation. Various defense methods have proposed image-to-image mapping methods, either including these perturbations in the training process or removing them in a preprocessing denoising step. In doing so, existing methods often ignore that the natural RGB images in today's datasets are not captured but, in fact, recovered from RAW color filter a…
MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations
School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK（生物医学工程与成像科学学院，伦敦国王学院，伦敦，英国）；deepc GMBH, Munich, Germany（deepc 德国慕尼黑分公司）
Learned Lightweight Smartphone ISP with Unpaired Data
Faculty of Computer Science Alexandru Ioan Cuza University of Iasi（亚历山德鲁·伊奥安·库扎大学计算机科学系）；Computer Vision Lab, CAIDAS & IFI University of Würzburg（弗赖堡大学计算机视觉实验室）
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Image signal processors (ISPs) are historically grown legacy software systems for reconstructing color images from noisy raw sensor measurements. They are usually composited of many heuristic blocks for denoising, demosaicking, and color restoration. Color reproduction in this context is of particular importance, since the raw colors are often severely distorted, and each smart phone manufacturer has developed their own characteristic heuristics for improving the color rendition, for example of skin tones and other visually important colors. In recent years there has been strong interest in re…
CAP-UDF: Learning Unsigned Distance Functions Progressively from Raw Point Clouds with Consistency-Aware Field Optimization
School of Software, Tsinghua University（清华大学软件学院）；Center for Artificial Intelligence and Robotics, New York University Abu Dhabi（纽约大学阿布扎比分校人工智能与机器人中心）；Wayne State University（韦恩州立大学）
Pacific Northwest National Lab (PNNL)（西北太平洋国家实验室）；University of Washington (UW)（华盛顿大学）
Deep learning algorithms excel at extracting patterns from raw data, and with large datasets, they have been very successful in computer vision and natural language applications. However, in other domains, large datasets on which to learn representations from may not exist. In this work, we develop a novel multimodal CNN-MLP neural network architecture that utilizes both domain-specific feature engineering as well as learned representations from raw data. We illustrate the effectiveness of such network designs in the chemical sciences, for predicting biodegradability. DeepBioD, a multimodal CN…
JPEG images can be further compressed to enhance the storage and transmission of large-scale image datasets. Existing learned lossless compressors for RGB images cannot be well transferred to JPEG images due to the distinguishing distribution of DCT coefficients and raw pixels. In this paper, we propose a novel framework for learned lossless compression of JPEG images that achieves end-to-end optimized prediction of the distribution of decoded DCT coefficients. To enable learning in the frequency domain, DCT coefficients are partitioned into groups to utilize implicit local redundancy. An auto…
Advances in photo editing and manipulation tools have made it significantly easier to create fake imagery. Learning to detect such manipulations, however, remains a challenging problem due to the lack of sufficient amounts of manipulated training data. In this paper, we propose a learning algorithm for detecting visual image manipulations that is trained only using a large dataset of real photographs. The algorithm uses the automatically recorded photo EXIF metadata as supervisory signal for training a model to determine whether an image is self-consistent -- that is, whether its content could…
Self-Supervised AI-Generated Image Detection: A Camera Metadata Perspective
Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；Department of Computer Science, Fudan University（复旦大学计算机科学系）；School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院）；Department of Computer Science and the Shenzhen Research Institute, City University of Hong Kong（城市大学计算机科学系和深圳研究院）
In this paper, we explore neural network models that learn to associate segments of spoken audio captions with the semantically relevant portions of natural images that they refer to. We demonstrate that these audio-visual associative localizations emerge from network-internal representations learned as a by-product of training to perform an image-audio retrieval task. Our models operate directly on the image pixels and speech waveform, and do not rely on any conventional supervision in the form of labels, segmentations, or alignments between the modalities during training. We perform analysis…
Janelia Farm Research Campus（珍妮莉亚农场研究园区）；Howard Hughes Medical Institute（霍华德·休斯医学研究所）
For image recognition and labeling tasks, recent results suggest that machine learning methods that rely on manually specified feature representations may be outperformed by methods that automatically derive feature representations based on the data. Yet for problems that involve analysis of 3d objects, such as mesh segmentation, shape retrieval, or neuron fragment agglomeration, there remains a strong reliance on hand-designed feature descriptors. In this paper, we evaluate a large set of hand-designed 3d feature descriptors alongside features learned from the raw data using both end-to-end a…
ISPDiffuser: Learning RAW-to-sRGB Mappings with Texture-Aware Diffusion Models and Histogram-Guided Color Consistency
School of Aeronautics and Astronautics, Sichuan University（四川大学航空航天学院）；University of Electronic Science and Technology of China（电子科技大学）；Megvii Technology（旷视科技）
GraphDerm: Fusing Imaging, Physical Scale, and Metadata in a Population-Graph Classifier for Dermoscopic Lesions
School of Computer Science, Institute for Research in Fundamental Sciences (IPM)（计算机科学学院，基础科学研究所（IPM））；Department of Physics, Shahid Beheshti University（物理系，肖德·贝赫谢什提大学）；Faculty of Science, University of Tehran（科学学院，德黑兰大学）；Fatemeh Zahra Hospital, Alborz University of Medical Sciences（法特梅·扎拉医院，阿尔bor兹医学科学大学）；Department of Electrical and Computer Engineering, Sungkyunkwan University（电气与计算机工程系，成均馆大学）；Department of Computer Science and Engineering, Shahid Beheshti University（计算机科学与工程系，肖德·贝赫谢什提大学）；Autoimmune Bullous Diseases Research Center, Razi Hospital, Tehran University of Medical Sciences（自身免疫性水疱病研究中心，拉齐医院，德黑兰医学科学大学）
KTH - Royal Institute of Technology（瑞典皇家理工学院）；Elekta AB（医科达公司）
We propose the Learned Primal-Dual algorithm for tomographic reconstruction. The algorithm accounts for a (possibly non-linear) forward operator in a deep neural network by unrolling a proximal primal-dual optimization method, but where the proximal operators have been replaced with convolutional neural networks. The algorithm is trained end-to-end, working directly from raw measured data and it does not depend on any initial reconstruction such as FBP. We compare performance of the proposed method on low dose CT reconstruction against FBP, TV, and deep learning based post-processing of FBP. F…
