Robotics applications in urban environments are subject to obstacles that exhibit specular reflections hampering autonomous navigation. On the other hand, these reflections are highly polarized and this extra information can successfully be used to segment the specular areas. In nature, polarized light is obtained by reflection or scattering. Deep Convolutional Neural Networks (DCNNs) have shown excellent segmentation results, but require a significant amount of data to achieve best performances. The lack of data is usually overcomed by using augmentation methods. However, unlike RGB images, polarization images are not only scalar (intensity) images and standard augmentation techniques cannot be applied straightforwardly. We propose to enhance deep learning models through a regularized augmentation procedure applied to polarimetric data in order to characterize scenes more effectively under challenging conditions. We subsequently observe an average of 18.1% improvement in IoU between non augmented and regularized training procedures on real world data.
Normandie Univ（诺曼底大学）；UNIROUEN（鲁昂大学）；UNIHAVRE（勒阿弗尔大学）；INSA Rouen（鲁昂国立应用科学学院）；University of Burgundy（勃艮第大学）；UBFC（勃艮第-弗朗什孔泰大学）
Polarimetric imaging, along with deep learning, has shown improved performances on different tasks including scene analysis. However, its robustness may be questioned because of the small size of the training datasets. Though the issue could be solved by data augmentation, polarization modalities are subject to physical feasibility constraints unaddressed by classical data augmentation techniques. To address this issue, we propose to use CycleGAN, an image translation technique based on deep generative models that solely relies on unpaired data, to transfer large labeled road scene datasets to…
The approaches for analyzing the polarimetric scattering matrix of polarimetric synthetic aperture radar (PolSAR) data have always been the focus of PolSAR image classification. Generally, the polarization coherent matrix and the covariance matrix obtained by the polarimetric scattering matrix only show a limited number of polarimetric information. In order to solve this problem, we propose a sparse scattering coding way to deal with polarimetric scattering matrix and obtain a close complete feature. This encoding mode can also maintain polarimetric information of scattering matrix completely.…
POLSAR image has an advantage over optical image because it can be acquired independently of cloud cover and solar illumination. PolSAR image classification is a hot and valuable topic for the interpretation of POLSAR image. In this paper, a novel POLSAR image classification method is proposed based on polarimetric scattering coding and sparse support matrix machine. First, we transform the original POLSAR data to get a real value matrix by the polarimetric scattering coding, which is called polarimetric scattering matrix and is a sparse matrix. Second, the sparse support matrix machine is use…
Key Lab for Information Science of Electromagnetic Waves (MoE), Fudan University（复旦大学电磁波信息科学教育部重点实验室）
A deep neural networks based method is proposed to convert single polarization grayscale SAR image to fully polarimetric. It consists of two components: a feature extractor network to extract hierarchical multi-scale spatial features of grayscale SAR image, followed by a feature translator network to map spatial feature to polarimetric feature with which the polarimetric covariance matrix of each pixel can be reconstructed. Both qualitative and quantitative experiments with real fully polarimetric data are conducted to show the efficacy of the proposed method. The reconstructed full-pol SAR im…
For polarimetric SAR (PolSAR) image classification, it is a challenge to classify the aggregated terrain types, such as the urban area, into semantic homogenous regions due to sharp bright-dark variations in intensity. The aggregated terrain type is formulated by the similar ground objects aggregated together. In this paper, a polarimetric hierarchical semantic model (PHSM) is firstly proposed to overcome this disadvantage based on the constructions of a primal-level and a middle-level semantic. The primal-level semantic is a polarimetric sketch map which consists of sketch segments as the spa…
Rutgers, The State University of New Jersey（罗格斯大学，新泽西州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）
The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face recognition quite a challenging problem for both human-examiners and computer vision algorithms. Previous approaches utilize a two-step procedure (visible feature estimation and visible image reconstruction) to synthesize the visible image given the corresponding polarimetric thermal image. However, these are regarded as two disjoint steps and hence may hinder the performance of visible face reconstruction. We argue that joint optimization would be a better w…
Army Academy of Artillery and Air Defense, PLA（中国人民解放军陆军炮兵防空兵学院）；Anhui Provincial Key Laboratory of Polarization Imaging Detection Technology（安徽省偏振成像探测技术重点实验室）；School of Internet, Anhui University（安徽大学互联网学院）；Anhui Institute of Optics and Fine Mechanics, Chinese Academy of Sciences（中国科学院安徽光学精密机械研究所）
Pixel-wise operations between polarimetric images are important for processing polarization information. For the lack of such operations, the polarization information cannot be fully utilized in convolutional neural network(CNN). In this paper, a novel end-to-end CNN framework for polarization vision tasks is proposed, which enables the networks to take full advantage of polarimetric images. The framework consists of two sub-networks: a polarization-parameter-constructing network (PPCN) and a task network. PPCN implements pixel-wise operations between images in the CNN form with 1x1 convolutio…
Thermal to visible face verification is a challenging problem due to the large domain discrepancy between the modalities. Existing approaches either attempt to synthesize visible faces from thermal faces or extract robust features from these modalities for cross-modal matching. In this paper, we take a different approach in which we make use of the attributes extracted from the visible image to synthesize the attribute-preserved visible image from the input thermal image for cross-modal matching. A pre-trained VGG-Face network is used to extract the attributes from the visible image. Then, a n…
Johns Hopkins University（约翰斯·霍普金斯大学）；U.S. Army CCDC Army Research Laboratory（美国陆军CCDC陆军研究实验室）
Polarimetric thermal to visible face verification entails matching two images that contain significant domain differences. Several recent approaches have attempted to synthesize visible faces from thermal images for cross-modal matching. In this paper, we take a different approach in which rather than focusing only on synthesizing visible faces from thermal faces, we also propose to synthesize thermal faces from visible faces. Our intuition is based on the fact that thermal images also contain some discriminative information about the person for verification. Deep features from a pre-trained C…
The data fusion technology aims to aggregate the characteristics of different data and obtain products with multiple data advantages. To solves the problem of reduced resolution of PolSAR images due to system limitations, we propose a fully polarimetric synthetic aperture radar (PolSAR) images and single-polarization synthetic aperture radar SAR (SinSAR) images fusion network to generate high-resolution PolSAR (HR-PolSAR) images. To take advantage of the polarimetric information of the low-resolution PolSAR (LR-PolSAR) image and the spatial information of the high-resolution single-polarizatio…
Rutgers, The State University of New Jersey（罗格斯大学新泽西州立大学）；U.S. Army Research Laboratory（美国陆军研究实验室）；Booz Allen Hamilton（博思艾伦汉密尔顿公司）；Johns Hopkins University（约翰斯·霍普金斯大学）
The large domain discrepancy between faces captured in polarimetric (or conventional) thermal and visible domain makes cross-domain face verification a highly challenging problem for human examiners as well as computer vision algorithms. Previous approaches utilize either a two-step procedure (visible feature estimation and visible image reconstruction) or an input-level fusion technique, where different Stokes images are concatenated and used as a multi-channel input to synthesize the visible image given the corresponding polarimetric signatures. Although these methods have yielded improvemen…
University of Bristol（布里斯托大学）；Shanghai Em-Data Technology Co., Ltd.（上海数慧科技股份有限公司）；China University of Mining and Technology（中国矿业大学）；University of Derby（德比大学）
Polarimetric synthetic aperture radar (PolSAR) image segmentation is currently of great importance in image processing for remote sensing applications. However, it is a challenging task due to two main reasons. Firstly, the label information is difficult to acquire due to high annotation costs. Secondly, the speckle effect embedded in the PolSAR imaging process remarkably degrades the segmentation performance. To address these two issues, we present a contextual PolSAR image semantic segmentation method in this paper.With a newly defined channelwise consistent feature set as input, the three-d…
Beijing University of Posts and Telecommunications（北京邮电大学）；Xiong’an Aerospace Information Research Institute（雄安航空航天信息研究所）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学多媒体信息处理国家重点实验室，计算机科学学院）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（视觉技术国家工程研究中心，计算机科学学院）
In this work, we propose a novel method for the detailed reconstruction of transparent objects by exploiting polarimetric cues. Most of the existing methods usually lack sufficient constraints and suffer from the over-smooth problem. Hence, we introduce polarization information as a complementary cue. We implicitly represent the object's geometry as a neural network, while the polarization render is capable of rendering the object's polarization images from the given shape and illumination configuration. Direct comparison of the rendered polarization images to the real-world captured images wi…
Instituto de Computação, Universidade Federal de Alagoas（阿拉戈斯联邦大学计算研究所）；Departamento de Computación, Universidad de Buenos Aires（布宜诺斯艾利斯大学计算系）
We present an approach for polarimetric Synthetic Aperture Radar (SAR) image region boundary detection based on the use of B-Spline active contours and a new model for polarimetric SAR data: the GHP distribution. In order to detect the boundary of a region, initial B-Spline curves are specified, either automatically or manually, and the proposed algorithm uses a deformable contours technique to find the boundary. In doing this, the parameters of the polarimetric GHP model for the data are estimated, in order to find the transition points between the region being segmented and the surrounding a…
In this paper, we introduce an iterative speckle filtering method for polarimetric SAR (PolSAR) images based on the bilateral filter. To locally adapt to the spatial structure of images, this filter relies on pixel similarities in both spatial and radiometric domains. To deal with polarimetric data, we study the use of similarities based on a statistical distance called Kullback-Leibler divergence as well as two geodesic distances on Riemannian manifolds. To cope with speckle, we propose to progressively refine the result thanks to an iterative scheme. Experiments are run over synthetic and ex…
Riemannian Complex Hermit Positive Definite Convolution Network for Polarimetric SAR Image Classification
Department of Computer Science and Technology, Shaanxi Key Laboratory for Network Computing and Security Technology, Xi’an University of Technology（计算机科学与技术系、网络计算与安全技术陕西省重点实验室、西安理工大学）；School of Artificial Intelligence, Xidian University（人工智能学院、西安电子科技大学）
Polarimetric synthetic aperture radar (PolSAR) images are widely used in disaster detection and military reconnaissance and so on. However, their interpretation faces some challenges, e.g., deficiency of labeled data, inadequate utilization of data information and so on. In this paper, a complex-valued generative adversarial network (GAN) is proposed for the first time to address these issues. The complex number form of model complies with the physical mechanism of PolSAR data and in favor of utilizing and retaining amplitude and phase information of PolSAR data. GAN architecture and semi-supe…
Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)（德国航空航天中心（DLR）遥感技术研究所（IMF））；Signal Processing in Earth Observation (SiPEO), Technical University of Munich (TUM)（慕尼黑工业大学（TUM）地球观测信号处理实验室（SiPEO））
Multi-modal data fusion has recently been shown promise in classification tasks in remote sensing. Optical data and radar data, two important yet intrinsically different data sources, are attracting more and more attention for potential data fusion. It is already widely known that, a machine learning based methodology often yields excellent performance. However, the methodology relies on a large training set, which is very expensive to achieve in remote sensing. The semi-supervised manifold alignment (SSMA), a multi-modal data fusion algorithm, has been designed to amplify the impact of an exi…
University of Hong Kong（香港大学）；Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）
Although deep learning has achieved great success in image classification tasks, its performance is subject to the quantity and quality of training samples. For classification of polarimetric synthetic aperture radar (PolSAR) images, it is nearly impossible to annotate the images from visual interpretation. Therefore, it is urgent for remote sensing scientists to develop new techniques for PolSAR image classification under the condition of very few training samples. In this letter, we take the advantage of active learning and propose active ensemble deep learning (AEDL) for PolSAR image classi…
As a representation learning method, nearest regularized subspace(NRS) algorithm is an effective tool to obtain both accuracy and speed for PolSAR image classification. However, existing NRS methods use the polarimetric feature vector but the PolSAR original covariance matrix(known as Hermitian positive definite(HPD)matrix) as the input. Without considering the matrix structure, existing NRS-based methods cannot learn correlation among channels. How to utilize the original covariance matrix to NRS method is a key problem. To address this limit, a Riemannian NRS method is proposed, which consid…
In this paper, we explore and compare multiple solutions to the problem of data augmentation in image classification. Previous work has demonstrated the effectiveness of data augmentation through simple techniques, such as cropping, rotating, and flipping input images. We artificially constrain our access to data to a small subset of the ImageNet dataset, and compare each data augmentation technique in turn. One of the more successful data augmentations strategies is the traditional transformations mentioned above. We also experiment with GANs to generate images of different styles. Finally, w…
MapBox；Level5 Engineering, Lyft Inc（Lyft公司Level5工程部门）；University of Michigan（密歇根大学）
Data augmentation is a commonly used technique for increasing both the size and the diversity of labeled training sets by leveraging input transformations that preserve output labels. In computer vision domain, image augmentations have become a common implicit regularization technique to combat overfitting in deep convolutional neural networks and are ubiquitously used to improve performance. While most deep learning frameworks implement basic image transformations, the list is typically limited to some variations and combinations of flipping, rotating, scaling, and cropping. Moreover, the ima…
This work presents an analysis of the efficiency of image augmentations for the face recognition problem from limited data. We considered basic manipulations, generative methods, and their combinations for augmentations. Our results show that augmentations, in general, can considerably improve the quality of face recognition systems and the combination of generative and basic approaches performs better than the other tested techniques.
Tel-Aviv University（特拉维夫大学）；The Interdisciplinary Center Herzliya（赫兹利亚跨学科研究中心）
Image augmentation techniques apply transformation functions such as rotation, shearing, or color distortion on an input image. These augmentations were proven useful in improving neural networks' generalization ability. In this paper, we present a novel augmentation operation, InAugment, that exploits image internal statistics. The key idea is to copy patches from the image itself, apply augmentation operations on them, and paste them back at random positions on the same image. This method is simple and easy to implement and can be incorporated with existing augmentation techniques. We test I…
PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model
the BRain and Artificial INtelligence Lab (BRAIN LAB), School of Automation, Northwestern Polytechnical University（人工智能实验室（BRAIN LAB）、自动化学院、西北工业大学）；the Shenzhen Research Institute of Northwestern Polytechnical University（西北工业大学深圳研究院）；the National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室、计算机科学学院、北京大学）；the National Key Laboratory of Microwave Imaging Technology, Chinese Academy of Sciences（微波成像技术国家重点实验室、中国科学院）；the Aerospace Information Research Institute, Chinese Academy of Sciences（航空信息研究所、中国科学院）
Weakly-supervised land classification for coastal zone based on deep convolutional neural networks by incorporating dual-polarimetric characteristics into training dataset
In this work, we exploit convolutional neural networks (CNNs) for the classification of very high resolution (VHR) polarimetric SAR (PolSAR) data. Due to the significant appearance of heterogeneous textures within these data, not only polarimetric features but also structural tensors are exploited to feed CNN models. For deep networks, we use the SegNet model for semantic segmentation, which corresponds to pixelwise classification in remote sensing. Our experiments on the airborne F-SAR data show that for VHR PolSAR images, SegNet could provide high accuracy for the classification task; and in…
Semantic image segmentation aims to obtain object labels with precise boundaries, which usually suffers from overfitting. Recently, various data augmentation strategies like regional dropout and mix strategies have been proposed to address the problem. These strategies have proved to be effective for guiding the model to attend on less discriminative parts. However, current strategies operate at the image level, and objects and the background are coupled. Thus, the boundaries are not well augmented due to the fixed semantic scenario. In this paper, we propose ObjectAug to perform object-level …
Despite the large augmentation family, only a few cherry-picked robust augmentation policies are beneficial to self-supervised image representation learning. In this paper, we propose a directional self-supervised learning paradigm (DSSL), which is compatible with significantly more augmentations. Specifically, we adapt heavy augmentation policies after the views lightly augmented by standard augmentations, to generate harder view (HV). HV usually has a higher deviation from the original image than the lightly augmented standard view (SV). Unlike previous methods equally pairing all augmented …
Automated data augmentation, which aims at engineering augmentation policy automatically, recently draw a growing research interest. Many previous auto-augmentation methods utilized a Density Matching strategy by evaluating policies in terms of the test-time augmentation performance. In this paper, we theoretically and empirically demonstrated the inconsistency between the train and validation set of small-scale medical image datasets, referred to as in-domain sampling bias. Next, we demonstrated that the in-domain sampling bias might cause the inefficiency of Density Matching. To address the …
This paper introduces Augraphy, a Python library for constructing data augmentation pipelines which produce distortions commonly seen in real-world document image datasets. Augraphy stands apart from other data augmentation tools by providing many different strategies to produce augmented versions of clean document images that appear as if they have been altered by standard office operations, such as printing, scanning, and faxing through old or dirty machines, degradation of ink over time, and handwritten markings. This paper discusses the Augraphy tool, and shows how it can be used both as a…
Guangdong Provincial People’s Hospital（广东省人民医院）；Guangdong Academy of Medical Sciences（广东省医学科学院）；Guangdong Cardiovascular Institute（广东省心血管病研究所）；Wuhan University（武汉大学）；Guangzhou Women and Children’s Medical Center（广州市妇女儿童医疗中心）；Guangzhou Medical University（广州医科大学）
Background: Medical images are more difficult to acquire and annotate than natural images, which results in data augmentation technologies often being used in medical image segmentation tasks. Most data augmentation technologies used in medical segmentation were originally developed on natural images and do not take into account the characteristic that the overall layout of medical images is standard and fixed. Methods: Based on the characteristics of medical images, we developed the cutting-splicing data augmentation (CS-DA) method, a novel data augmentation technology for medical image segme…
This paper proposes a novel learning based high-dynamic-range (HDR) reconstruction method using a polarization camera. We utilize a previous observation that polarization filters with different orientations can attenuate natural light differently, and we treat the multiple images acquired by the polarization camera as a set acquired under different exposure times, to introduce the development of solutions for the HDR reconstruction problem. We propose a deep HDR reconstruction framework with a feature masking mechanism that uses polarimetric cues available from the polarization camera, called …
Polarimetric Synthetic Aperture Radar (PolSAR) images are establishing as an important source of information in remote sensing applications. The most complete format this type of imaging produces consists of complex-valued Hermitian matrices in every image coordinate and, as such, their visualization is challenging. They also suffer from speckle noise which reduces the signal-to-noise ratio. Smoothing techniques have been proposed in the literature aiming at preserving different features and, analogously, projections from the cone of Hermitian positive matrices to different color representatio…
This paper is concerned with specular reflection removal based on tensor low-rank decomposition framework with the help of polarization information. Our method is motivated by the observation that the specular highlight of an image is sparsely distributed while the remaining diffuse reflection can be well approximated by a linear combination of several distinct colors using a low-rank and sparse decomposition framework. Unlike current solutions, our tensor low-rank decomposition keeps the spatial structure of specular and diffuse information which enables us to recover the diffuse image under …
Polarimetric SAR data has the characteristics of all-weather, all-time and so on, which is widely used in many fields. However, the data of annotation is relatively small, which is not conducive to our research. In this paper, we have collected five open polarimetric SAR images, which are images of the San Francisco area. These five images come from different satellites at different times, which has great scientific research value. We annotate the collected images at the pixel level for image classification and segmentation. For the convenience of researchers, the annotated data is open source…
Diffusion-based Virtual Staining from Polarimetric Mueller Matrix Imaging
The Hong Kong University of Science and Technology（香港科技大学）；Beijing Institute of Collaborative Innovation（北京协同创新研究院）；Peking University Health Science Center（北京大学医学部）；University of Oxford（牛津大学）；Peking University（北京大学）
This paper is concerned with polarimetric dense map reconstruction based on a polarization camera with the help of relative depth information as a prior. In general, polarization imaging is able to reveal information about surface normal such as azimuth and zenith angles, which can support the development of solutions to the problem of dense reconstruction, especially in texture-poor regions. However, polarimetric shape cues are ambiguous due to two types of polarized reflection (specular/diffuse). Although methods have been proposed to address this issue, they either are offline and therefore…
Beihang University（北京航空航天大学）；The University of Tokyo（东京大学）；Peng Cheng Laboratory（鹏城实验室）
The emergence of the single-chip polarized color sensor now allows for simultaneously capturing chromatic and polarimetric information of the scene on a monochromatic image plane. However, unlike the usual camera with an embedded demosaicing method, the latest polarized color camera is not delivered with an in-built demosaicing tool. For demosaicing, the users have to down-sample the captured images or to use traditional interpolation techniques. Neither of them can perform well since the polarization and color are interdependent. Therefore, joint chromatic and polarimetric demosaicing is the …
Polarisation Filter Array (PFA) cameras allow the analysis of light polarisation state in a simple and cost-effective manner. Such filter arrays work as the Bayer pattern for colour cameras, sharing similar advantages and drawbacks. Among the others, the raw image must be demosaiced considering the local variations of the PFA and the characteristics of the imaged scene. Non-linear effects, like the cross-talk among neighbouring pixels, are difficult to explicitly model and suggest the potential advantage of a data-driven learning approach. However, the PFA cannot be removed from the sensor, ma…
Physics-Aware Complex-Valued State Space Model with Scattering-Prior Feature Modulation for PolSAR Image Classification
The College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）；The School of Information Science and Engineering, Yunnan University（云南大学信息科学与工程学院）；The Department of Signal Theory and Communications, Polytechnic University of Catalonia（加泰罗尼亚理工大学信号理论与通信系）；The School of Information and Cyberspace Security, Ningxia University（宁夏大学信息与网络空间安全学院）
