Domain gaps of sensor modalities pose a challenge for the design of autonomous robots. Taking a step towards closing this gap, we propose two unsupervised training frameworks for finding a common representation of LiDAR and camera data. The first method utilizes a double Siamese training structure to ensure consistency in the results. The second method uses a Canny edge image guiding the networks towards a desired representation. All networks are trained in an unsupervised manner, leaving room for scalability. The results are evaluated using common computer vision applications, and the limitations of the proposed approaches are outlined.
Tracking vehicles in LIDAR point clouds is a challenging task due to the sparsity of the data and the dense search space. The lack of structure in point clouds impedes the use of convolution filters usually employed in 2D object tracking. In addition, structuring point clouds is cumbersome and implies losing fine-grained information. As a result, generating proposals in 3D space is expensive and inefficient. In this paper, we leverage the dense and structured Bird Eye View (BEV) representation of LIDAR point clouds to efficiently search for objects of interest. We use an efficient Region Propo…
Shanghai Jiao Tong University（上海交通大学）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）
Crowd segmentation is a fundamental task serving as the basis of crowded scene analysis, and it is highly desirable to obtain refined pixel-level segmentation maps. However, it remains a challenging problem, as existing approaches either require dense pixel-level annotations to train deep learning models or merely produce rough segmentation maps from optical or particle flows with physical models. In this paper, we propose the Motion Prior-Aware Siamese Network (MPASNET) for unsupervised crowd semantic segmentation. This model not only eliminates the need for annotation but also yields high-qu…
Nanjing University of Posts and Telecommunications（南京邮电大学）；Nanjing University of Science and Technology（南京理工大学）
Robust road detection is a key challenge in safe autonomous driving. Recently, with the rapid development of 3D sensors, more and more researchers are trying to fuse information across different sensors to improve the performance of road detection. Although many successful works have been achieved in this field, methods for data fusion under deep learning framework is still an open problem. In this paper, we propose a Siamese deep neural network based on FCN-8s to detect road region. Our method uses data collected from a monocular color camera and a Velodyne-64 LiDAR sensor. We project the LiD…
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Point clouds are challenging to process due to their sparsity, therefore autonomous vehicles rely more on appearance attributes than pure geometric features. However, 3D LIDAR perception can provide crucial information for urban navigation in challenging light or weather conditions. In this paper, we investigate the versatility of Shape Completion for 3D Object Tracking in LIDAR point clouds. We design a Siamese tracker that encodes model and candidate shapes into a compact latent representation. We regularize the encoding by enforcing the latent representation to decode into an object model s…
Nanjing University of Science and Technology（南京理工大学）；PCA Lab（PCA实验室）
Extensive research efforts have been dedicated to deep learning based odometry. Nonetheless, few efforts are made on the unsupervised deep lidar odometry. In this paper, we design a novel framework for unsupervised lidar odometry with the IMU, which is never used in other deep methods. First, a pair of siamese LSTMs are used to obtain the initial pose from the linear acceleration and angular velocity of IMU. With the initial pose, we perform the rigid transform on the current frame and align it closer to the last frame. Then, we extract vertex and normal features from the transformed point clo…
Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）
Unsupervised visual anomaly detection conveys practical significance in many scenarios and is a challenging task due to the unbounded definition of anomalies. Moreover, most previous methods are application-specific, and establishing a unified model for anomalies across application scenarios remains unsolved. This paper proposes a novel hybrid framework termed Siamese Transition Masked Autoencoders(ST-MAE) to handle various visual anomaly detection tasks uniformly via deep feature transition. Concretely, the proposed method first extracts hierarchical semantics features from a pre-trained deep…
Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
This paper presents Dense Siamese Network (DenseSiam), a simple unsupervised learning framework for dense prediction tasks. It learns visual representations by maximizing the similarity between two views of one image with two types of consistency, i.e., pixel consistency and region consistency. Concretely, DenseSiam first maximizes the pixel level spatial consistency according to the exact location correspondence in the overlapped area. It also extracts a batch of region embeddings that correspond to some sub-regions in the overlapped area to be contrasted for region consistency. In contrast t…
High-precision lidar odomety is an essential part of autonomous driving. In recent years, deep learning methods have been widely used in lidar odomety tasks, but most of the current methods only extract the global features of the point clouds. It is impossible to obtain more detailed point-level features in this way. In addition, only the fully connected layer is used to estimate the pose. The fully connected layer has achieved obvious results in the classification task, but the changes in pose are a continuous rather than discrete process, high-precision pose estimation can not be obtained on…
Inception Institute of Artificial Intelligence（人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）；Australian National University（澳大利亚国立大学）
We introduce a novel network, called CO-attention Siamese Network (COSNet), to address the unsupervised video object segmentation task from a holistic view. We emphasize the importance of inherent correlation among video frames and incorporate a global co-attention mechanism to improve further the state-of-the-art deep learning based solutions that primarily focus on learning discriminative foreground representations over appearance and motion in short-term temporal segments. The co-attention layers in our network provide efficient and competent stages for capturing global correlations and sce…
Scene understanding is a pivotal task for autonomous vehicles to safely navigate in the environment. Recent advances in deep learning enable accurate semantic reconstruction of the surroundings from LiDAR data. However, these models encounter a large domain gap while deploying them on vehicles equipped with different LiDAR setups which drastically decreases their performance. Fine-tuning the model for every new setup is infeasible due to the expensive and cumbersome process of recording and manually labeling new data. Unsupervised Domain Adaptation (UDA) techniques are thus essential to fill t…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；SenseTime Research（商汤科技研究院）；The University of Sydney（悉尼大学）；Peng Cheng Laboratory（鹏城实验室）；Shanghai AI Laboratory（上海人工智能实验室）
Unsupervised learning has been popular in various computer vision tasks, including visual object tracking. However, prior unsupervised tracking approaches rely heavily on spatial supervision from template-search pairs and are still unable to track objects with strong variation over a long time span. As unlimited self-supervision signals can be obtained by tracking a video along a cycle in time, we investigate evolving a Siamese tracker by tracking videos forward-backward. We present a novel unsupervised tracking framework, in which we can learn temporal correspondence both on the classificatio…
Northwestern Polytechnical University（西北工业大学）；Australian National University（澳大利亚国立大学）；ACRV；Data61 CSIRO（联邦科学与工业研究组织Data61）；NEC Laboratories America（NEC美国实验室）
In this paper, we present LidarStereoNet, the first unsupervised Lidar-stereo fusion network, which can be trained in an end-to-end manner without the need of ground truth depth maps. By introducing a novel "Feedback Loop'' to connect the network input with output, LidarStereoNet could tackle both noisy Lidar points and misalignment between sensors that have been ignored in existing Lidar-stereo fusion studies. Besides, we propose to incorporate a piecewise planar model into network learning to further constrain depths to conform to the underlying 3D geometry. Extensive quantitative and qualit…
Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operat…
NTT Human Informatics Laboratories（NTT人类信息学实验室）；NTT Communication Science Laboratories（NTT通信科学实验室）
Intrinsic image decomposition (IID) is the task that decomposes a natural image into albedo and shade. While IID is typically solved through supervised learning methods, it is not ideal due to the difficulty in observing ground truth albedo and shade in general scenes. Conversely, unsupervised learning methods are currently underperforming supervised learning methods since there are no criteria for solving the ill-posed problems. Recently, light detection and ranging (LiDAR) is widely used due to its ability to make highly precise distance measurements. Thus, we have focused on the utilization…
Recently, learning-based ego-motion estimation approaches have drawn strong interest from studies mostly focusing on visual perception. These groundbreaking works focus on unsupervised learning for odometry estimation but mostly for visual sensors. Compared to images, a learning-based approach using Light Detection and Ranging (LiDAR) has been reported in a few studies where, most often, a supervised learning framework is proposed. In this paper, we propose a novel approach to geometry-aware deep LiDAR odometry trainable via both supervised and unsupervised frameworks. We incorporate the Itera…
Robust data association is necessary for virtually every SLAM system and finding corresponding points is typically a preprocessing step for scan alignment algorithms. Traditionally, handcrafted feature descriptors were used for these problems but recently learned descriptors have been shown to perform more robustly. In this work, we propose a local feature descriptor for 3D LiDAR scans. The descriptor is learned using a Convolutional Neural Network (CNN). Our proposed architecture consists of a Siamese network for learning a feature descriptor and a metric learning network for matching the des…
The rapid development of embedded hardware in autonomous vehicles broadens their computational capabilities, thus bringing the possibility to mount more complete sensor setups able to handle driving scenarios of higher complexity. As a result, new challenges such as multiple detections of the same object have to be addressed. In this work, a siamese network is integrated into the pipeline of a well-known 3D object detector approach to suppress duplicate proposals coming from different cameras via re-identification. Additionally, associations are exploited to enhance the 3D box regression of th…
Early Autism Diagnosis based on Path Signature and Siamese Unsupervised Feature Compressor
University of Pennsylvania（宾夕法尼亚大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；The Affiliated Shenzhen School of Guangdong Experimental High School（广东实验中学附属深圳学校）；Pazhou Lab（琶洲实验室）
Telekooperation Lab, Technical University Darmstadt（达姆施塔特工业大学远程协作实验室）
In this work, we address the problem of unsupervised moving object segmentation (MOS) in 4D LiDAR data recorded from a stationary sensor, where no ground truth annotations are involved. Deep learning-based state-of-the-art methods for LiDAR MOS strongly depend on annotated ground truth data, which is expensive to obtain and scarce in existence. To close this gap in the stationary setting, we propose a novel 4D LiDAR representation based on multivariate time series that relaxes the problem of unsupervised MOS to a time series clustering problem. More specifically, we propose modeling the change…
In this paper, we focus on a less explored, but more realistic and complex problem of domain adaptation in LiDAR semantic segmentation. There is a significant drop in performance of an existing segmentation model when training (source domain) and testing (target domain) data originate from different LiDAR sensors. To overcome this shortcoming, we propose an unsupervised domain adaptation framework that leverages unlabeled target domain data for self-supervision, coupled with an unpaired mask transfer strategy to mitigate the impact of domain shifts. Furthermore, we introduce the gated adapter …
TREND: Unsupervised 3D Representation Learning via Temporal Forecasting for LiDAR Perception
The University of Hong Kong（香港大学）；Yale University（耶鲁大学）；Shanghai AI Laboratory（上海人工智能实验室）；HKU Shanghai Intelligent Computing Research Center（香港大学上海智能计算研究中心）
Rotation is among the long prevailing, yet still unresolved, hard challenges encountered in visual object tracking. The existing deep learning-based tracking algorithms use regular CNNs that are inherently translation equivariant, but not designed to tackle rotations. In this paper, we first demonstrate that in the presence of rotation instances in videos, the performance of existing trackers is severely affected. To circumvent the adverse effect of rotations, we present rotation-equivariant Siamese networks (RE-SiamNets), built through the use of group-equivariant convolutional layers compris…
With the convenient availability of remote sensing data, how to make models to interpret complex remote sensing data attracts wide attention. In remote sensing data, hyperspectral images contain spectral information and LiDAR contains elevation information. Hence, more explorations are warranted to better fuse the features of different source data. In this paper, we introduce semantic understanding to dynamically fuse data from two different sources, extract features of HSI and LiDAR through different capsule network branches and improve self-supervised loss and random rigid rotation in Canoni…
Depth perception is pivotal in many fields, such as robotics and autonomous driving, to name a few. Consequently, depth sensors such as LiDARs rapidly spread in many applications. The 3D point clouds generated by these sensors must often be coupled with an RGB camera to understand the framed scene semantically. Usually, the former is projected over the camera image plane, leading to a sparse depth map. Unfortunately, this process, coupled with the intrinsic issues affecting all the depth sensors, yields noise and gross outliers in the final output. Purposely, in this paper, we propose an effec…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））
Despite its importance, unsupervised domain adaptation (UDA) on LiDAR semantic segmentation is a task that has not received much attention from the research community. Only recently, a completion-based 3D method has been proposed to tackle the problem and formally set up the adaptive scenarios. However, the proposed pipeline is complex, voxel-based and requires multi-stage inference, which inhibits it for real-time inference. We propose a range image-based, effective and efficient method for solving UDA on LiDAR segmentation. The method exploits class prototypes from the source domain to pseud…
SenseTime Research（商汤科技研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Center for Research on Intelligent Perception and Computing CASIA（中国科学院自动化研究所智能感知与计算研究中心）；Center for Excellence in Brain Science and Intelligence Technology CAS（中国科学院脑科学与智能技术卓越创新中心）
Despite the importance of unsupervised object detection, to the best of our knowledge, there is no previous work addressing this problem. One main issue, widely known to the community, is that object boundaries derived only from 2D image appearance are ambiguous and unreliable. To address this, we exploit LiDAR clues to aid unsupervised object detection. By exploiting the 3D scene structure, the issue of localization can be considerably mitigated. We further identify another major issue, seldom noticed by the community, that the long-tailed and open-ended (sub-)category distribution should be …
ShanghaiTech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；The Chinese University of Hong Kong（香港中文大学）
Domain adaptation for Cross-LiDAR 3D detection is challenging due to the large gap on the raw data representation with disparate point densities and point arrangements. By exploring domain-invariant 3D geometric characteristics and motion patterns, we present an unsupervised domain adaptation method that overcomes above difficulties. First, we propose the Spatial Geometry Alignment module to extract similar 3D shape geometric features of the same object class to align two domains, while eliminating the effect of distinct point distributions. Second, we present Temporal Motion Alignment module …
Wuhan University（武汉大学）；Finnish Geospatial Research Institute（芬兰地理空间研究所）；Hubei University of Technology（湖北工业大学）
As an important technology in 3D mapping, autonomous driving, and robot navigation, LiDAR odometry is still a challenging task. Appropriate data structure and unsupervised deep learning are the keys to achieve an easy adjusted LiDAR odometry solution with high performance. Utilizing compact 2D structured spherical ring projection model and voxel model which preserves the original shape of input data, we propose a fully unsupervised Convolutional Auto-Encoder based LiDAR Odometry (CAE-LO) that detects interest points from spherical ring data using 2D CAE and extracts features from multi-resolut…
University of Trento（特伦托大学）；LTCI, Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院LTCI实验室）；Huawei Research（华为研究院）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）；Sapienza University of Rome（罗马第一大学）
3D object detectors based only on LiDAR point clouds hold the state-of-the-art on modern street-view benchmarks. However, LiDAR-based detectors poorly generalize across domains due to domain shift. In the case of LiDAR, in fact, domain shift is not only due to changes in the environment and in the object appearances, as for visual data from RGB cameras, but is also related to the geometry of the point clouds (e.g., point density variations). This paper proposes SF-UDA$^{3D}$, the first Source-Free Unsupervised Domain Adaptation (SF-UDA) framework to domain-adapt the state-of-the-art PointRCNN …
Indian Institute of Information Technology, Sri City（斯里城印度信息技术学院）
Deep learning has shown a great improvement in the performance of visual tasks. Image retrieval is the task of extracting the visually similar images from a database for a query image. The feature matching is performed to rank the images. Various hand-designed features have been derived in past to represent the images. Nowadays, the power of deep learning is being utilized for automatic feature learning from data in the field of biomedical image analysis. Autoencoder and Siamese networks are two deep learning models to learn the latent space (i.e., features or embedding). Autoencoder works bas…
Energy-saving LIDAR camera for short distances estimates an object's distance using temporally intensity-coded laser light pulses and calculates the maximum correlation with the back-scattered pulse. Though on low power, the backs-scattered pulse is noisy and unstable, which leads to inaccurate and unreliable depth estimation. To address this problem, we use GANs (Generative Adversarial Networks), which are two neural networks that can learn complicated class distributions through an adversarial process. We learn the LIDAR camera's hidden properties and behavior, creating a novel, fully unsupe…
Valeo Egypt AI Research（法雷奥埃及人工智能研究院）；Cairo University（开罗大学）
Data scarcity is a bottleneck to machine learning-based perception modules, usually tackled by augmenting real data with synthetic data from simulators. Realistic models of the vehicle perception sensors are hard to formulate in closed form, and at the same time, they require the existence of paired data to be learned. In this work, we propose two unsupervised neural sensor models based on unpaired domain translations with CycleGANs and Neural Style Transfer techniques. We employ CARLA as the simulation environment to obtain simulated LiDAR point clouds, together with their annotations for dat…
Very-high-resolution (VHR) images can provide abundant ground details and spatial geometric information. Change detection in multi-temporal VHR images plays a significant role in urban expansion and area internal change analysis. Nevertheless, traditional change detection methods can neither take full advantage of spatial context information nor cope with the complex internal heterogeneity of VHR images. In this paper, a powerful feature extraction model entitled multi-scale feature convolution unit (MFCU) is adopted for change detection in multi-temporal VHR images. MFCU can extract multi-sca…
Nanjing University of Science and Technology（南京理工大学）
3D object tracking in point clouds is still a challenging problem due to the sparsity of LiDAR points in dynamic environments. In this work, we propose a Siamese voxel-to-BEV tracker, which can significantly improve the tracking performance in sparse 3D point clouds. Specifically, it consists of a Siamese shape-aware feature learning network and a voxel-to-BEV target localization network. The Siamese shape-aware feature learning network can capture 3D shape information of the object to learn the discriminative features of the object so that the potential target from the background in sparse po…
CO^3: Cooperative Unsupervised 3D Representation Learning for Autonomous Driving
The University of Hong Kong（香港大学）；Shanghai AI Laboratory（上海人工智能实验室）；Huawei Noah’s Ark Lab（华为诺亚实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）
Automatic extraction of buildings in urban scenes has become a subject of growing interest in the domain of photogrammetry and remote sensing, particularly with the emergence of LiDAR systems since mid-1990s. However, in reality, this task is still very challenging due to the complexity of building size and shapes, as well as its surrounding environment. Active contour model, colloquially called snake model, which has been extensively used in many applications in computer vision and image processing, is also applied to extract buildings from aerial/satellite imagery. Motivated by the limitatio…
Baidu Research（百度研究院）；National Engineering Laboratory of Deep Learning Technology and Application, China（中国深度学习技术与应用国家工程实验室）；University of Macau（澳门大学）；University of Technology Sydney（悉尼科技大学）；Beijing Institute of Technology（北京理工大学）
Existing approaches for unsupervised point cloud pre-training are constrained to either scene-level or point/voxel-level instance discrimination. Scene-level methods tend to lose local details that are crucial for recognizing the road objects, while point/voxel-level methods inherently suffer from limited receptive field that is incapable of perceiving large objects or context environments. Considering region-level representations are more suitable for 3D object detection, we devise a new unsupervised point cloud pre-training framework, called ProposalContrast, that learns robust 3D representa…
National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Alibaba Group（阿里巴巴集团）
Recent self-supervised contrastive learning methods greatly benefit from the Siamese structure that aims at minimizing distances between positive pairs. For high performance Siamese representation learning, one of the keys is to design good contrastive pairs. Most previous works simply apply random sampling to make different crops of the same image, which overlooks the semantic information that may degrade the quality of views. In this work, we propose ContrastiveCrop, which could effectively generate better crops for Siamese representation learning. Firstly, a semantic-aware object localizati…
Multi-Correlation Siamese Transformer Network with Dense Connection for 3D Single Object Tracking
Zhengzhou University（郑州大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nullmax Inc（Nullmax公司）
