Electrical and Computer Engineering and ASRI, Seoul National University（首尔国立大学电气与计算机工程系及自动化系统研究所）；Electrical Engineering and Computer Science, University of California at Merced（加州大学默塞德分校电气工程与计算机科学系）
Single image reflection separation is an ill-posed problem since two scenes, a transmitted scene and a reflected scene, need to be inferred from a single observation. To make the problem tractable, in this work we assume that categories of two scenes are known. It allows us to address the problem by generating both scenes that belong to the categories while their contents are constrained to match with the observed image. A novel network architecture is proposed to render realistic images of both scenes based on adversarial learning. The network can be trained in a weakly supervised manner, i.e., it learns to separate an observed image without corresponding ground truth images of transmission and reflection scenes which are difficult to collect in practice. Experimental results on real and synthetic datasets demonstrate that the proposed algorithm performs favorably against existing methods.
Single Image Reflection Separation via Dual Prior Interaction Transformer
School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；School of Computer Science and Technology, Guangdong University of Technology（广东工业大学计算机科学与技术学院）；Shenzhen Key Laboratory of Visual Object Detection and Recognition, Harbin Institute of Technology（哈尔滨工业大学深圳视觉目标检测与识别重点实验室）
Unified Removal of Raindrops and Reflections: A New Benchmark and A Novel Pipeline
School of Aeronautics and Astronautics, Zhejiang University（浙江大学航空航天学院）；Huanjiang Laboratory（浣江实验室）；Hangzhou Institute of Technology, Xidian University（西安电子科技大学杭州研究院）；The Chinese University of Hong Kong（香港中文大学）
School of Computer Science, Peking University（北京大学计算机学院）；National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology（视觉技术国家工程研究中心）；AI Innovation Center（人工智能创新中心）
Beihang University（北京航空航天大学）；Peng Cheng Laboratory（鹏城实验室）；The University of Tokyo（东京大学）
Since specular reflection often exists in the real captured images and causes deviation between the recorded color and intrinsic color, specular reflection separation can bring advantages to multiple applications that require consistent object surface appearance. However, due to the color of an object is significantly influenced by the color of the illumination, the existing researches still suffer from the near-duplicate challenge, that is, the separation becomes unstable when the illumination color is close to the surface color. In this paper, we derive a polarization guided model to incorpo…
We present an approach to separating reflection from a single image. The approach uses a fully convolutional network trained end-to-end with losses that exploit low-level and high-level image information. Our loss function includes two perceptual losses: a feature loss from a visual perception network, and an adversarial loss that encodes characteristics of images in the transmission layers. We also propose a novel exclusion loss that enforces pixel-level layer separation. We create a dataset of real-world images with reflection and corresponding ground-truth transmission layers for quantitati…
In this paper, we address the problem of reflection removal and deblurring from a single image captured by a plenoptic camera. We develop a two-stage approach to recover the scene depth and high resolution textures of the reflected and transmitted layers. For depth estimation in the presence of reflections, we train a classifier through convolutional neural networks. For recovering high resolution textures, we assume that the scene is composed of planar regions and perform the reconstruction of each layer by using an explicit form of the plenoptic camera point spread function. The proposed fra…
We introduce the eigentask framework for lifelong learning. An eigentask is a pairing of a skill that solves a set of related tasks, paired with a generative model that can sample from the skill's input space. The framework extends generative replay approaches, which have mainly been used to avoid catastrophic forgetting, to also address other lifelong learning goals such as forward knowledge transfer. We propose a wake-sleep cycle of alternating task learning and knowledge consolidation for learning in our framework, and instantiate it for lifelong supervised learning and lifelong RL. We achi…
Photographs taken through a glass surface often contain an approximately linear superposition of reflected and transmitted layers. Decomposing an image into these layers is generally an ill-posed task and the use of an additional image prior and user provided cues is presently necessary in order to obtain good results. Current annotation approaches rely on a strong sparsity assumption. For images with significant texture this assumption does not typically hold, thus rendering the annotation process unviable. In this paper we show that using a Gaussian Mixture Model patch prior, the correct loc…
Taking pictures through glass windows almost always produces undesired reflections that degrade the quality of the photo. The ill-posed nature of the reflection removal problem reached the attention of many researchers for more than decades. The main challenge of this problem is the lack of real training data and the necessity of generating realistic synthetic data. In this paper, we proposed a single image reflection removal method based on context understanding modules and adversarial training to efficiently restore the transmission layer without reflection. We also propose a complex data ge…
Reflections often degrade the quality of the image by obstructing the background scene. This is not desirable for everyday users, and it negatively impacts the performance of multimedia applications that process images with reflections. Most current methods for removing reflections utilize supervised-learning models. However, these models require an extensive number of image pairs to perform well, especially on natural images with reflection, which is difficult to achieve in practice. In this paper, we propose a novel unsupervised framework for single-image reflection separation. Instead of le…
We propose a novel reflection color model consisting of body essence and (mixed) neuter, and present an effective method for separating dichromatic reflection components using a single image. Body essence is an entity invariant to interface reflection, and has two degrees of freedom unlike hue and maximum chromaticity. As a result, the proposed method is insensitive to noise and proper for colors around CMY (cyan, magenta, and yellow) as well as RGB (red, green, and blue), contrary to the maximum chromaticity-based methods. Interface reflection is separated by using a Gaussian function, which …
Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；State Key Laboratory of Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室和视觉技术国家工程研究中心）
This paper is concerned with specular reflection removal based on tensor low-rank decomposition framework with the help of polarization information. Our method is motivated by the observation that the specular highlight of an image is sparsely distributed while the remaining diffuse reflection can be well approximated by a linear combination of several distinct colors using a low-rank and sparse decomposition framework. Unlike current solutions, our tensor low-rank decomposition keeps the spatial structure of specular and diffuse information which enables us to recover the diffuse image under …
Single image reflection separation (SIRS), as a representative blind source separation task, aims to recover two layers, $\textit{i.e.}$, transmission and reflection, from one mixed observation, which is challenging due to the highly ill-posed nature. Existing deep learning based solutions typically restore the target layers individually, or with some concerns at the end of the output, barely taking into account the interaction across the two streams/branches. In order to utilize information more efficiently, this work presents a general yet simple interactive strategy, namely $\textit{your tr…
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Qihoo 360 Artificial Intelligence Institute（奇虎360人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）
Style synthesis attracts great interests recently, while few works focus on its dual problem "style separation". In this paper, we propose the Style Separation and Synthesis Generative Adversarial Network (S3-GAN) to simultaneously implement style separation and style synthesis on object photographs of specific categories. Based on the assumption that the object photographs lie on a manifold, and the contents and styles are independent, we employ S3-GAN to build mappings between the manifold and a latent vector space for separating and synthesizing the contents and styles. The S3-GAN consists …
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））
Recently, deep learning-based single image reflection separation methods have been exploited widely. To benefit the learning approach, a large number of training image pairs (i.e., with and without reflections) were synthesized in various ways, yet they are away from a physically-based direction. In this paper, physically based rendering is used for faithfully synthesizing the required training images, and a corresponding network structure and loss term are proposed. We utilize existing RGBD/RGB images to estimate meshes, then physically simulate the light transportation between meshes, glass,…
RL-RIG: A Generative Spatial Reasoner via Intrinsic Reflection
Zhiyuan College, Shanghai Jiao Tong University（上海交通大学紫元学院）；Huazhong University of Science and Technology（华中科技大学）；National University of Singapore（新加坡国立大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）
Reflections in videos are obstructions that often occur when videos are taken behind reflective surfaces like glass. These reflections reduce the quality of such videos, lead to information loss and degrade the accuracy of many computer vision algorithms. A video containing reflections is a combination of background and reflection layers. Thus, reflection removal is equivalent to decomposing the video into two layers. This, however, is a challenging and ill-posed problem as there is an infinite number of valid decompositions. To address this problem, we propose a user-assisted method for video…
Multiple stains are usually used to highlight biological substances in biomedical image analysis. To decompose multiple stains for co-localization quantification, blind source separation is usually performed. Prior model-based stain separation methods usually rely on stains' spatial distributions over an image and may fail to solve the co-localization problem. With the advantage of machine learning, deep generative models are used for this purpose. Since prior knowledge of imaging models is ignored in purely data-driven solutions, these methods may be sub-optimal. In this study, a novel learni…
University of Delaware（特拉华大学）；ShanghaiTech University（上海科技大学）；Microsoft Research（微软研究院）
We propose a novel approach that jointly removes reflection or translucent layer from a scene and estimates scene depth. The input data are captured via light field imaging. The problem is couched as minimizing the rank of the transmitted scene layer via Robust Principle Component Analysis (RPCA). We also impose regularization based on piecewise smoothness, gradient sparsity, and layer independence to simultaneously recover 3D geometry of the transmitted layer. Experimental results on synthetic and real data show that our technique is robust and reliable, and can handle a broad range of layer …
Beihang University（北京航空航天大学）；International Digital Economy Academy（国际数字经济学院）；University of Amsterdam（阿姆斯特丹大学）；Peng Cheng Laboratory（鹏城实验室）
Reflection is common in images capturing scenes behind a glass window, which is not only a disturbance visually but also influence the performance of other computer vision algorithms. Single image reflection removal is an ill-posed problem because the color at each pixel needs to be separated into two values, i.e., the desired clear background and the reflection. To solve it, existing methods propose priors such as smoothness, color consistency. However, the low-level priors are not reliable in complex scenes, for instance, when capturing a real outdoor scene through a window, both the foregro…
Shandong University（山东大学）；Microsoft Research（微软研究院）；Shenzhen Research Institute, Shandong University（山东大学深圳研究院）
This paper proposes a deep neural network structure that exploits edge information in addressing representative low-level vision tasks such as layer separation and image filtering. Unlike most other deep learning strategies applied in this context, our approach tackles these challenging problems by estimating edges and reconstructing images using only cascaded convolutional layers arranged such that no handcrafted or application-specific image-processing components are required. We apply the resulting transferrable pipeline to two different problem domains that are both sensitive to edges, nam…
State Key Laboratory of Virtual Reality Technology and Systems（虚拟现实技术与系统全国重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机学院）
Many real world vision tasks, such as reflection removal from a transparent surface and intrinsic image decomposition, can be modeled as single image layer separation. However, this problem is highly ill-posed, requiring accurately aligned and hard to collect triplet data to train the CNN models. To address this problem, this paper proposes an unsupervised method that requires no ground truth data triplet in training. At the core of the method are two assumptions about data distributions in the latent spaces of different layers, based on which a novel unsupervised layer separation pipeline can…
Beijing Institute of Technology（北京理工大学）；Australian National University（澳大利亚国立大学）；Yale-NUS College（耶鲁-新加坡国立大学学院）
This paper deals with a challenging, frequently encountered, yet not properly investigated problem in two-frame optical flow estimation. That is, the input frames are compounds of two imaging layers -- one desired background layer of the scene, and one distracting, possibly moving layer due to transparency or reflection. In this situation, the conventional brightness constancy constraint -- the cornerstone of most existing optical flow methods -- will no longer be valid. In this paper, we propose a robust solution to this problem. The proposed method performs both optical flow estimation, and …
A Lightweight Deep Exclusion Unfolding Network for Single Image Reflection Removal
College of Computer Science and Technology, National University of Defense Technology（国防科技大学计算机科学与技术学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机与信息工程学院）；Department of Electrical and Electronic Engineering, Imperial College London（伦敦帝国理工学院电气与电子工程系）
Harbin Institute of Technology at Shenzhen（哈尔滨工业大学（深圳））；Tencent（腾讯）；The Chinese University of Hong Kong at Shenzhen（香港中文大学（深圳））
Restoring the clean background from the superimposed images containing a noisy layer is the common crux of a classical category of tasks on image restoration such as image reflection removal, image deraining and image dehazing. These tasks are typically formulated and tackled individually due to the diverse and complicated appearance patterns of noise layers within the image. In this work we present the Deep-Masking Generative Network (DMGN), which is a unified framework for background restoration from the superimposed images and is able to cope with different types of noise. Our proposed DMGN…
PolarGuide-GSDR: 3D Gaussian Splatting Driven by Polarization Priors and Deferred Reflection for Real-World Reflective Scenes
North China University of Technology（中国北方工业大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；University of Science and Technology Beijing（北京科技大学）
University of Westminster（威斯敏斯特大学）；Calico Life Sciences LLC（Calico生命科学有限责任公司）
Dixon MRI is widely used for body composition studies. Current processing methods associated with large whole-body volumes are time intensive and prone to artifacts during fat-water separation performed on the scanner, making the data difficult to analyse. The most common artifact are fat-water swaps, where the labels are inverted at the voxel level. It is common for researchers to discard swapped data (generally around 10%), which can be wasteful and lead to unintended biases. The UK Biobank is acquiring Dixon MRI for over 100,000 participants, and thousands of swaps will occur. If those go u…
The Hong Kong Polytechnic University（香港理工大学）；ETH Zurich（苏黎世联邦理工学院）；Center for Artificial Intelligence and Robotics, HKISI-CAS（中国科学院香港智能科学研究所人工智能与机器人中心）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
When imaging through a semi-reflective medium such as glass, the reflection of another scene can often be found in the captured images. It degrades the quality of the images and affects their subsequent analyses. In this paper, a novel deep neural network approach for solving the reflection problem in imaging is presented. Traditional reflection removal methods not only require long computation time for solving different optimization functions, their performance is also not guaranteed. As array cameras are readily available in nowadays imaging devices, we first suggest in this paper a multiple…
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；Center for Applied Scientific Computing（应用科学计算中心）
Solving inverse problems continues to be a challenge in a wide array of applications ranging from deblurring, image inpainting, source separation etc. Most existing techniques solve such inverse problems by either explicitly or implicitly finding the inverse of the model. The former class of techniques require explicit knowledge of the measurement process which can be unrealistic, and rely on strong analytical regularizers to constrain the solution space, which often do not generalize well. The latter approaches have had remarkable success in part due to deep learning, but require a large coll…
