Intrinsic image decomposition aims to factorize an image into albedo (reflectance) and shading (illumination) sub-components. Being ill-posed and under-constrained, it is a very challenging computer vision problem. There are infinite pairs of reflectance and shading images that can reconstruct the same input. To address the problem, Intrinsic Images in the Wild provides an optimization framework based on a dense conditional random field (CRF) formulation that considers long-range material relations. We improve upon their model by introducing illumination invariant image descriptors: color ratios. The color ratios and the reflectance intrinsic are both invariant to illumination and thus are highly correlated. Through detailed experiments, we provide ways to inject the color ratios into the dense CRF optimization. Our approach is physics-based, learning-free and leads to more accurate and robust reflectance decompositions.
University of Würzburg（维尔茨堡大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）
Separating an image into reflectance and shading layers poses a challenge for learning approaches because no large corpus of precise and realistic ground truth decompositions exists. The Intrinsic Images in the Wild~(IIW) dataset provides a sparse set of relative human reflectance judgments, which serves as a standard benchmark for intrinsic images. A number of methods use IIW to learn statistical dependencies between the images and their reflectance layer. Although learning plays an important role for high performance, we show that a standard signal processing technique achieves performance o…
Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by unknown light. Existing solutions alleviate the ambiguity by either explicitly associating reflectance to light conditions or resolving light conditions in a supervised manner. This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner. The key idea is to represent the reflectance as four neural intrinsics fields, i.e., position, light, specular, and shadow, based on which the neural light field is implicitly associated with light clue…
We propose a novel intrinsic image decomposition network considering reflectance consistency. Intrinsic image decomposition aims to decompose an image into illumination-invariant and illumination-variant components, referred to as ``reflectance'' and ``shading,'' respectively. Although there are three consistencies that the reflectance and shading should satisfy, most conventional work does not sufficiently account for consistency with respect to reflectance, owing to the use of a white-illuminant decomposition model and the lack of training images capturing the same objects under various illu…
University of Amsterdam（阿姆斯特丹大学）；DUniversum B.V.（3DUniversum B.V.）
Semantic segmentation of outdoor scenes is problematic when there are variations in imaging conditions. It is known that albedo (reflectance) is invariant to all kinds of illumination effects. Thus, using reflectance images for semantic segmentation task can be favorable. Additionally, not only segmentation may benefit from reflectance, but also segmentation may be useful for reflectance computation. Therefore, in this paper, the tasks of semantic segmentation and intrinsic image decomposition are considered as a combined process by exploring their mutual relationship in a joint fashion. To th…
Beihang University（北京航空航天大学）；Tencent（腾讯）；University of Amsterdam（阿姆斯特丹大学）；Peng Cheng Laboratory（鹏城实验室）
Intrinsic image decomposition, which is an essential task in computer vision, aims to infer the reflectance and shading of the scene. It is challenging since it needs to separate one image into two components. To tackle this, conventional methods introduce various priors to constrain the solution, yet with limited performance. Meanwhile, the problem is typically solved by supervised learning methods, which is actually not an ideal solution since obtaining ground truth reflectance and shading for massive general natural scenes is challenging and even impossible. In this paper, we propose a nove…
University of Amsterdam（阿姆斯特丹大学）；D Universum（3D宇宙公司）
In general, intrinsic image decomposition algorithms interpret shading as one unified component including all photometric effects. As shading transitions are generally smoother than reflectance (albedo) changes, these methods may fail in distinguishing strong photometric effects from reflectance variations. Therefore, in this paper, we propose to decompose the shading component into direct (illumination) and indirect shading (ambient light and shadows) subcomponents. The aim is to distinguish strong photometric effects from reflectance variations. An end-to-end deep convolutional neural networ…
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
Intrinsic image decomposition is a severely under-constrained problem. User interactions can help to reduce the ambiguity of the decomposition considerably. The traditional way of user interaction is to draw scribbles that indicate regions with constant reflectance or shading. However the effect scopes of the scribbles are quite limited, so dozens of scribbles are often needed to rectify the whole decomposition, which is time consuming. In this paper we propose an efficient way of user interaction that users need only to annotate the color composition of the image. Color composition reveals th…
School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；Department of Electrical and Electronic Engineering, University of Cagliari（卡利亚里大学电气与电子工程系）
Face presentation attacks have become a major threat to face recognition systems and many countermeasures have been proposed in the past decade. However, most of them are devoted to 2D face presentation attacks, rather than 3D face masks. Unlike the real face, the 3D face mask is usually made of resin materials and has a smooth surface, resulting in reflectance differences. So, we propose a novel detection method for 3D face mask presentation attack by modeling reflectance differences based on intrinsic image analysis. In the proposed method, the face image is first processed with intrinsic im…
A fundamental problem in computer vision is that of inferring the intrinsic, 3D structure of the world from flat, 2D images of that world. Traditional methods for recovering scene properties such as shape, reflectance, or illumination rely on multiple observations of the same scene to overconstrain the problem. Recovering these same properties from a single image seems almost impossible in comparison -- there are an infinite number of shapes, paint, and lights that exactly reproduce a single image. However, certain explanations are more likely than others: surfaces tend to be smooth, paint ten…
Intrinsic image decomposition is the process of separating the reflectance and shading layers of an image, which is a challenging and underdetermined problem. In this paper, we propose to systematically address this problem using a deep convolutional neural network (CNN). Although deep learning (DL) has been recently used to handle this application, the current DL methods train the network only on synthetic images as obtaining ground truth reflectance and shading for real images is difficult. Therefore, these methods fail to produce reasonable results on real images and often perform worse tha…
Stony Brook University（石溪大学）；Computer Vision Center（计算机视觉中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）
Automatic document content processing is affected by artifacts caused by the shape of the paper, non-uniform and diverse color of lighting conditions. Fully-supervised methods on real data are impossible due to the large amount of data needed. Hence, the current state of the art deep learning models are trained on fully or partially synthetic images. However, document shadow or shading removal results still suffer because: (a) prior methods rely on uniformity of local color statistics, which limit their application on real-scenarios with complex document shapes and textures and; (b) synthetic …
University of Peradeniya（佩拉德尼亚大学）；Sri Lanka Technological Campus（斯里兰卡理工学院）
Intrinsic Image Decomposition is an open problem of generating the constituents of an image. Generating reflectance and shading from a single image is a challenging task specifically when there is no ground truth. There is a lack of unsupervised learning approaches for decomposing an image into reflectance and shading using a single image. We propose a neural network architecture capable of this decomposition using physics-based parameters derived from the image. Through experimental results, we show that (a) the proposed methodology outperforms the existing deep learning-based IID techniques …
Nanjing University（南京大学）；New York University（纽约大学）
Multispectral images contain many clues of surface characteristics of the objects, thus can be widely used in many computer vision tasks, e.g., recolorization and segmentation. However, due to the complex illumination and the geometry structure of natural scenes, the spectra curves of a same surface can look very different. In this paper, a Low Rank Multispectral Image Intrinsic Decomposition model (LRIID) is presented to decompose the shading and reflectance from a single multispectral image. We extend the Retinex model, which is proposed for RGB image intrinsic decomposition, for multispectr…
University of Amsterdam（阿姆斯特丹大学）；DUniversum（3D宇宙公司）
We investigate the use of photometric invariance and deep learning to compute intrinsic images (albedo and shading). We propose albedo and shading gradient descriptors which are derived from physics-based models. Using the descriptors, albedo transitions are masked out and an initial sparse shading map is calculated directly from the corresponding RGB image gradients in a learning-free unsupervised manner. Then, an optimization method is proposed to reconstruct the full dense shading map. Finally, we integrate the generated shading map into a novel deep learning framework to refine it and also…
Despite the high biometric performance, finger-vein recognition systems are vulnerable to presentation attacks (aka., spoofing attacks). In this paper, we present a new and robust approach for detecting presentation attacks on finger-vein biometric systems exploiting the 3D Shape (normal-map) and material properties (diffuse-map) of the finger. Observing the normal-map and diffuse-map exhibiting enhanced textural differences in comparison with the original finger-vein image, especially in the presence of varying illumination intensity, we propose to employ textural feature-descriptors on both …
University of Amsterdam（阿姆斯特丹大学）；DUniversum（3D宇宙公司）；AkzoNobel（阿克苏诺贝尔）
Intrinsic image decomposition (IID) is an under-constrained problem. Therefore, traditional approaches use hand crafted priors to constrain the problem. However, these constraints are limited when coping with complex scenes. Deep learning-based approaches learn these constraints implicitly through the data, but they often suffer from dataset biases (due to not being able to include all possible imaging conditions). In this paper, a combination of the two is proposed. Component specific priors like semantics and invariant features are exploited to obtain semantically and physically plausible re…
University of Amsterdam（阿姆斯特丹大学）；DUniversum（3DUniversum）
Intrinsic image decomposition is the process of recovering the image formation components (reflectance and shading) from an image. Previous methods employ either explicit priors to constrain the problem or implicit constraints as formulated by their losses (deep learning). These methods can be negatively influenced by strong illumination conditions causing shading-reflectance leakages. Therefore, in this paper, an end-to-end edge-driven hybrid CNN approach is proposed for intrinsic image decomposition. Edges correspond to illumination invariant gradients. To handle hard negative illumination t…
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
We address the problem of decomposing a single image into reflectance and shading. The difficulty comes from the fact that the components of image---the surface albedo, the direct illumination, and the ambient illumination---are coupled heavily in observed image. We propose to infer the shading by ordering pixels by their relative brightness, without knowing the absolute values of the image components beforehand. The pairwise shading orders are estimated in two ways: brightness order and low-order fittings of local shading field. The brightness order is a non-local measure, which can be applie…
Simon Fraser University（西蒙弗雷泽大学）；Microsoft Research（微软研究院）
We present an unsupervised approach for factorizing object appearance into highlight, shading, and albedo layers, trained by multi-view real images. To do so, we construct a multi-view dataset by collecting numerous customer product photos online, which exhibit large illumination variations that make them suitable for training of reflectance separation and can facilitate object-level decomposition. The main contribution of our approach is a proposed image representation based on local color distributions that allows training to be insensitive to the local misalignments of multi-view images. In…
Ghent University（根特大学）；Shenzhen Institute of Information Technology（深圳信息职业技术学院）；Beihang University（北京航空航天大学）
This paper presents a novel intrinsic image transfer (IIT) algorithm for illumination manipulation, which creates a local image translation between two illumination surfaces. This model is built on an optimization-based framework consisting of three photo-realistic losses defined on the sub-layers factorized by an intrinsic image decomposition. We illustrate that all losses can be reduced without the necessity of taking an intrinsic image decomposition under the well-known spatial-varying illumination illumination-invariant reflectance prior knowledge. Moreover, with a series of relaxations, a…
Single-image Reflectance and Transmittance Estimation from Any Flatbed Scanner
Politecnico di Milano（米兰理工大学）；Euro-Mediterranean Center on Climate Change (CMCC)（欧洲地中海气候变化中心（CMCC））；RFF-CMCC European Institute on Economics and the Environment (EIEE)（RFF-CMCC欧洲经济与环境研究所（EIEE））；Universidad Rey Juan Carlos（胡安卡洛斯国王大学）；Arquimea Research Center（阿基梅亚研究中心）；Adobe Research（奥多比研究中心）
Low-light Image Enhancement with Retinex Decomposition in Latent Space
School of Automation, Hangzhou Dianzi University（杭州电子科技大学自动化学院）；College of Artificial Intelligence, Jiaxing University（嘉兴大学人工智能学院）；School of Artificial Intelligence at Dalian University of Technology（大连理工大学人工智能学院）；Department of Computer Science at City University of Hong Kong（香港城市大学计算机科学系）
Shandong University（山东大学）；Microsoft Research（微软研究院）；Shenzhen Research Institute, Shandong University（山东大学深圳研究院）；Peking University（北京大学）
While invaluable for many computer vision applications, decomposing a natural image into intrinsic reflectance and shading layers represents a challenging, underdetermined inverse problem. As opposed to strict reliance on conventional optimization or filtering solutions with strong prior assumptions, deep learning based approaches have also been proposed to compute intrinsic image decompositions when granted access to sufficient labeled training data. The downside is that current data sources are quite limited, and broadly speaking fall into one of two categories: either dense fully-labeled im…
Northeastern University（东北大学）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）；Caliber I.D. Inc（Caliber I.D.公司）；SkinMedical Research and Diagnostics, P.L.L.C.（SkinMedical研究与诊断有限责任公司）；SUNY Downstate Medical Center（纽约州立大学下州医学中心）
We describe a new multiresolution "nested encoder-decoder" convolutional network architecture and use it to annotate morphological patterns in reflectance confocal microscopy (RCM) images of human skin for aiding cancer diagnosis. Skin cancers are the most common types of cancers, melanoma being the deadliest among them. RCM is an effective, non-invasive pre-screening tool for skin cancer diagnosis, with the required cellular resolution. However, images are complex, low-contrast, and highly variable, so that clinicians require months to years of expert-level training to be able to make accurat…
Informatics Institute（信息学研究所）；University of Amsterdam（阿姆斯特丹大学）
Most of the traditional work on intrinsic image decomposition rely on deriving priors about scene characteristics. On the other hand, recent research use deep learning models as in-and-out black box and do not consider the well-established, traditional image formation process as the basis of their intrinsic learning process. As a consequence, although current deep learning approaches show superior performance when considering quantitative benchmark results, traditional approaches are still dominant in achieving high qualitative results. In this paper, the aim is to exploit the best of the two …
Indian Statistical Institute（印度统计研究所）；National Institute of Technology, Rourkela（鲁尔凯拉国家技术学院）；Indian Institute of Information Technology, Sri City（斯里城印度信息技术学院）
Methods based on local image features have recently shown promise for texture classification tasks, especially in the presence of large intra-class variation due to illumination, scale, and viewpoint changes. Inspired by the theories of image structure analysis, this paper presents a simple, efficient, yet robust descriptor namely local jet pattern (LJP) for texture classification. In this approach, a jet space representation of a texture image is derived from a set of derivatives of Gaussian (DtGs) filter responses up to second order, so called local jet vectors (LJV), which also satisfy the …
SEDDI（SEDDI公司）；Universidad Rey Juan Carlos（胡安·卡洛斯国王大学）；Universidad Carlos III（卡洛斯三世大学）
Intrinsic imaging or intrinsic image decomposition has traditionally been described as the problem of decomposing an image into two layers: a reflectance, the albedo invariant color of the material; and a shading, produced by the interaction between light and geometry. Deep learning techniques have been broadly applied in recent years to increase the accuracy of those separations. In this survey, we overview those results in context of well-known intrinsic image data sets and relevant metrics used in the literature, discussing their suitability to predict a desirable intrinsic image decomposit…
Indian Institute of Information Technology, Lucknow（印度信息技术学院，勒克瑙）；Indian Institute of Information Technology, Allahabad（印度信息技术学院，安拉阿巴德）
Feature description is one of the most frequently studied areas in the expert systems and machine learning. Effective encoding of the images is an essential requirement for accurate matching. These encoding schemes play a significant role in recognition and retrieval systems. Facial recognition systems should be effective enough to accurately recognize individuals under intrinsic and extrinsic variations of the system. The templates or descriptors used in these systems encode spatial relationships of the pixels in the local neighbourhood of an image. Features encoded using these hand crafted d…
International Institute of Information Technology-Hyderabad（海得拉巴国际信息技术学院）
Intrinsic Image Decomposition (IID) is a challenging and interesting computer vision problem with various applications in several fields. We present novel semantic priors and an integrated approach for single image IID that involves analyzing image at three hierarchical context levels. Local context priors capture scene properties at each pixel within a small neighbourhood. Mid-level context priors encode object level semantics. Global context priors establish correspondences at the scene level. Our semantic priors are designed on both fixed and flexible regions, using selective search method …
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Mercedes Benz Research and Development India（梅赛德斯-奔驰印度研发中心）
We propose a new dataset for learning local image descriptors which can be used for significantly improved patch matching. Our proposed dataset consists of an order of magnitude more number of scenes, images, and positive and negative correspondences compared to the currently available Multi-View Stereo (MVS) dataset from Brown et al. The new dataset also has better coverage of the overall viewpoint, scale, and lighting changes in comparison to the MVS dataset. Our dataset also provides supplementary information like RGB patches with scale and rotations values, and intrinsic and extrinsic came…
École de Technologie Supérieure, University of Québec（魁北克大学高等技术学院）；Instituto Federal do Sul de Minas Gerais（南米纳斯吉拉斯联邦学院）；Pontificia Universidade Catolica do Paraná（巴拉那天主教大学）
Information from an image occurs over multiple and distinct spatial scales. Image pyramid multiresolution representations are a useful data structure for image analysis and manipulation over a spectrum of spatial scales. This paper employs the Gaussian-Laplacian pyramid to treat different spatial frequency bands of a texture separately. First, we generate three images corresponding to three levels of the Gaussian-Laplacian pyramid for an input image to capture intrinsic details. Then we aggregate features extracted from gray and color texture images using bio-inspired texture descriptors, info…
Intrinsic decomposition from a single image is a highly challenging task, due to its inherent ambiguity and the scarcity of training data. In contrast to traditional fully supervised learning approaches, in this paper we propose learning intrinsic image decomposition by explaining the input image. Our model, the Rendered Intrinsics Network (RIN), joins together an image decomposition pipeline, which predicts reflectance, shape, and lighting conditions given a single image, with a recombination function, a learned shading model used to recompose the original input based off of intrinsic image p…
Beneath the uncertain primitive visual features of face images are the primitive intrinsic structural patterns (PISP) essential for characterizing a sample face discriminative attributes. It is on this basis that this paper presents a simple yet effective facial descriptor formed from derivatives of Gaussian and Gabor Wavelets. The new descriptor is coined local edge gradient Gabor magnitude (LEGGM) pattern. LEGGM first uncovers the PISP locked in every pixel through determining the pixel gradient in relation to its neighbors using the Derivatives of Gaussians. Then, the resulting output is em…
InterLight: Leveraging Intrinsic Illumination Priors for Low-Light Image Enhancement
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University（武汉大学计算机学院多媒体软件国家工程研究中心）；Department of Computer Science, University of Macau（澳门大学计算机科学系）；Department of Computer Vision, Mohamed bin Zayed University of Artificial Intelligence（马尔代夫穆罕默德·本·扎耶德人工智能大学计算机视觉系）；School of Information Engineering, Guangdong University of Technology（广东技术大学信息工程学院）
Click to Grasp: Zero-Shot Precise Manipulation via Visual Diffusion Descriptors
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；Edinburgh Centre for Robotics（爱丁堡机器人中心）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
University of Oxford（牛津大学）；University of Jordan（约旦大学）
Intensity variations in image texture can provide powerful quantitative information about physical properties of biological tissue. However, tissue patterns can vary according to the utilized imaging system and are intrinsically correlated to the scale of analysis. In the case of ultrasound, the Nakagami distribution is a general model of the ultrasonic backscattering envelope under various scattering conditions and densities where it can be employed for characterizing image texture, but the subtle intra-heterogeneities within a given mass are difficult to capture via this model as it works at…
Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Pengcheng Laboratory（鹏城实验室）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
This paper considers the intra-image color-space of an object or a scene when these are subject to a dominant single-source of variation. The source of variation can be intrinsic or extrinsic (i.e., imaging conditions) to the object. We observe that the quantized colors for such objects typically lie on a planar subspace of RGB, and in some cases linear or polynomial curves on this plane are effective in capturing these color variations. We also observe that the inter-image color sub-spaces are robust as long as drastic illumination change is not involved. We illustrate the use of this analysi…
ShapeEmbed: a self-supervised learning framework for 2D contour quantification
European Bioinformatics Institute（欧洲生物信息学研究所）；European Molecular Biology Laboratory（欧洲分子生物学实验室）；University of Birmingham（伯明翰大学）；School of Computer Science（计算机科学学院）
Ecole Centrale de Lyon（里昂中央理工学院）；Université Lyon 1（里昂第一大学）；CNRS（法国国家科学研究中心）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
2D face analysis techniques, such as face landmarking, face recognition and face verification, are reasonably dependent on illumination conditions which are usually uncontrolled and unpredictable in the real world. An illumination robust preprocessing method thus remains a significant challenge in reliable face analysis. In this paper we propose a novel approach for improving lighting normalization through building the underlying reflectance model which characterizes interactions between skin surface, lighting source and camera sensor, and elaborates the formation of face color appearance. Spe…
The flare-productivity of an active region is observed to be related to its spatial complexity. Mount Wilson or McIntosh sunspot classifications measure such complexity but in a categorical way, and may therefore not use all the information present in the observations. Moreover, such categorical schemes hinder a systematic study of an active region's evolution for example. We propose fine-scale quantitative descriptors for an active region's complexity and relate them to the Mount Wilson classification. We analyze the local correlation structure within continuum and magnetogram data, as well a…
