Visual data in autonomous driving perception, such as camera image and LiDAR point cloud, can be interpreted as a mixture of two aspects: semantic feature and geometric structure. Semantics come from the appearance and context of objects to the sensor, while geometric structure is the actual 3D shape of point clouds. Most detectors on LiDAR point clouds focus only on analyzing the geometric structure of objects in real 3D space. Unlike previous works, we propose to learn both semantic feature and geometric structure via a unified multi-view framework. Our method exploits the nature of LiDAR scans -- 2D range images, and applies well-studied 2D convolutions to extract semantic features. By fusing semantic and geometric features, our method outperforms state-of-the-art approaches in all categories by a large margin. The methodology of combining semantic and geometric features provides a unique perspective of looking at the problems in real-world 3D point cloud detection.
Deep implicit surfaces excel at modeling generic shapes but do not always capture the regularities present in manufactured objects, which is something simple geometric primitives are particularly good at. In this paper, we propose a representation combining latent and explicit parameters that can be decoded into a set of deep implicit and geometric shapes that are consistent with each other. As a result, we can effectively model both complex and highly regular shapes that coexist in manufactured objects. This enables our approach to manipulate 3D shapes in an efficient and precise manner.
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；New York University（纽约大学）
Estimation of differential geometric quantities in discrete 3D data representations is one of the crucial steps in the geometry processing pipeline. Specifically, estimating normals and sharp feature lines from raw point cloud helps improve meshing quality and allows us to use more precise surface reconstruction techniques. When designing a learnable approach to such problems, the main difficulty is selecting neighborhoods in a point cloud and incorporating geometric relations between the points. In this study, we present a geometric attention mechanism that can provide such properties in a le…
Point cloud semantic segmentation from projected views, such as range-view (RV) and bird's-eye-view (BEV), has been intensively investigated. Different views capture different information of point clouds and thus are complementary to each other. However, recent projection-based methods for point cloud semantic segmentation usually utilize a vanilla late fusion strategy for the predictions of different views, failing to explore the complementary information from a geometric perspective during the representation learning. In this paper, we introduce a geometric flow network (GFNet) to explore th…
Bridging Domain Gap of Point Cloud Representations via Self-Supervised Geometric Augmentation
School of Computer Science, Nanjing University of Information Science & Technology（南京信息工程大学计算机科学与学院）；Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science & Technology（南京信息工程大学江苏省大气环境与装备技术协同创新中心）；School of Electronic and Information Engineering, South China University of Technology（华南理工大学电子与信息工程学院）；Peng Cheng Laboratory（鹏城实验室）；College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Bournemouth University（伯恩茅斯大学）；Alan Turing Institute（艾伦·图灵研究所）
Self-supervised learning is attracting large attention in point cloud understanding. However, exploring discriminative and transferable features still remains challenging due to their nature of irregularity and sparsity. We propose a geometrically and adaptively masked auto-encoder for self-supervised learning on point clouds, termed \textit{PointGame}. PointGame contains two core components: GATE and EAT. GATE stands for the geometrical and adaptive token embedding module; it not only absorbs the conventional wisdom of geometric descriptors that captures the surface shape effectively, but als…
University of Macau（澳门大学）；South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）
Existing deep learning algorithms for point cloud analysis mainly concern discovering semantic patterns from global configuration of local geometries in a supervised learning manner. However, very few explore geometric properties revealing local surface manifolds embedded in 3D Euclidean space to discriminate semantic classes or object parts as additional supervision signals. This paper is the first attempt to propose a unique multi-task geometric learning network to improve semantic analysis by auxiliary geometric learning with local shape properties, which can be either generated via physica…
MVTec Software GmbH（MVTec软件有限公司）；Technical University of Munich（慕尼黑工业大学）
We present a new method for the unsupervised detection of geometric anomalies in high-resolution 3D point clouds. In particular, we propose an adaptation of the established student-teacher anomaly detection framework to three dimensions. A student network is trained to match the output of a pretrained teacher network on anomaly-free point clouds. When applied to test data, regression errors between the teacher and the student allow reliable localization of anomalous structures. To construct an expressive teacher network that extracts dense local geometric descriptors, we introduce a novel self…
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lingnan University（岭南大学）；Hong Kong Metropolitan University（香港都会大学）
We propose GeoGCN, a novel geometric dual-domain graph convolution network for point cloud denoising (PCD). Beyond the traditional wisdom of PCD, to fully exploit the geometric information of point clouds, we define two kinds of surface normals, one is called Real Normal (RN), and the other is Virtual Normal (VN). RN preserves the local details of noisy point clouds while VN avoids the global shape shrinkage during denoising. GeoGCN is a new PCD paradigm that, 1) first regresses point positions by spatialbased GCN with the help of VNs, 2) then estimates initial RNs by performing Principal Comp…
Deep neural networks are prone to adversarial examples that maliciously alter the network's outcome. Due to the increasing popularity of 3D sensors in safety-critical systems and the vast deployment of deep learning models for 3D point sets, there is a growing interest in adversarial attacks and defenses for such models. So far, the research has focused on the semantic level, namely, deep point cloud classifiers. However, point clouds are also widely used in a geometric-related form that includes encoding and reconstructing the geometry. In this work, we are the first to consider the problem o…
EGP3D: Edge-guided Geometric Preserving 3D Point Cloud Super-resolution for RGB-D camera
School of Data Science, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）数据科学学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；City University of Hong Kong（香港城市大学）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Tsinghua University（清华大学）
Beihang University（北京航空航天大学）；The University of Sydney（悉尼大学）；The University of Hong Kong（香港大学）
Point cloud registration aims at estimating the geometric transformation between two point cloud scans, in which point-wise correspondence estimation is the key to its success. In addition to previous methods that seek correspondences by hand-crafted or learnt geometric features, recent point cloud registration methods have tried to apply RGB-D data to achieve more accurate correspondence. However, it is not trivial to effectively fuse the geometric and visual information from these two distinctive modalities, especially for the registration problem. In this work, we propose a new Geometry-Awa…
Delft University of Technology（代尔夫特理工大学）；Universitas Gadjah Mada（加查马达大学）
Learning from 3D point-cloud data has rapidly gained momentum, motivated by the success of deep learning on images and the increased availability of 3D~data. In this paper, we aim to construct anisotropic convolution layers that work directly on the surface derived from a point cloud. This is challenging because of the lack of a global coordinate system for tangential directions on surfaces. We introduce DeltaConv, a convolution layer that combines geometric operators from vector calculus to enable the construction of anisotropic filters on point clouds. Because these operators are defined on …
Geometric-Aware Hypergraph Reasoning for Novel Class Discovery in Point Cloud Segmentation
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University（人工智能学院、智能计算学院、天津大学）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院、合肥工业大学）；Department of Computer Science City St George’s, University of London（伦敦大学城市圣乔治学院计算机科学系）
University of Maryland, College Park（马里兰大学帕克分校）；Megvii Inc (Face++)（旷视科技（Face++））
Recent advances in deep convolutional neural networks (CNNs) have motivated researchers to adapt CNNs to directly model points in 3D point clouds. Modeling local structure has been proven to be important for the success of convolutional architectures, and researchers exploited the modeling of local point sets in the feature extraction hierarchy. However, limited attention has been paid to explicitly model the geometric structure amongst points in a local region. To address this problem, we propose Geo-CNN, which applies a generic convolution-like operation dubbed as GeoConv to each point and i…
Shanghai Jiao Tong University（上海交通大学）；East China Normal University（华东师范大学）；City University of Hong Kong（香港城市大学）；Xiamen University（厦门大学）
Boundary information plays a significant role in 2D image segmentation, while usually being ignored in 3D point cloud segmentation where ambiguous features might be generated in feature extraction, leading to misclassification in the transition area between two objects. In this paper, firstly, we propose a Boundary Prediction Module (BPM) to predict boundary points. Based on the predicted boundary, a boundary-aware Geometric Encoding Module (GEM) is designed to encode geometric information and aggregate features with discrimination in a neighborhood, so that the local features belonging to dif…
Web-based phishing attacks nowadays exploit popular cloud web hosting services and apps such as Google Sites and Typeform for hosting their attacks. Since these attacks originate from reputable domains and IP addresses of the cloud services, traditional phishing detection methods such as IP reputation monitoring and blacklisting are not very effective. Here we investigate the effectiveness of deep learning models in detecting this class of cloud-based phishing attacks. Specifically, we evaluate deep learning models for three phishing detection methods--LSTM model for URL analysis, YOLOv2 model…
With the popularity of 3D sensors in self-driving and other robotics applications, extensive research has focused on designing novel neural network architectures for accurate 3D point cloud completion. However, unlike in point cloud classification and reconstruction, the role of adversarial samples in3D point cloud completion has seldom been explored. In this work, we show that training with adversarial samples can improve the performance of neural networks on 3D point cloud completion tasks. We propose a novel approach to generate adversarial samples that benefit both the performance of clean…
Technical University of Munich (TUM)（慕尼黑工业大学）；University of Applied Science (HM)（应用科学大学（HM））；Bruno Kessler Foundation (FBK)（布鲁诺·凯斯勒基金会（FBK））
Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models' façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to iden…
Zero-shot point cloud segmentation by transferring geometric primitives
The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；Inceptio（嬴彻科技）；ShanghaiTech University（上海科技大学）；Texas A&M University（德州农工大学）
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）
In this paper, we investigate the combination of voxel-based methods and point-based methods, and propose a novel end-to-end two-stage 3D object detector named SGNet for point clouds scenes. The voxel-based methods voxelize the scene to regular grids, which can be processed with the current advanced feature learning frameworks based on convolutional layers for semantic feature learning. Whereas the point-based methods can better extract the geometric feature of the point due to the coordinate reservations. The combination of the two is an effective solution for 3D object detection from point c…
Stanford University（斯坦福大学）；Facebook AI Research（脸书人工智能研究院）
Fitting geometric primitives to 3D point cloud data bridges a gap between low-level digitized 3D data and high-level structural information on the underlying 3D shapes. As such, it enables many downstream applications in 3D data processing. For a long time, RANSAC-based methods have been the gold standard for such primitive fitting problems, but they require careful per-input parameter tuning and thus do not scale well for large datasets with diverse shapes. In this work, we introduce Supervised Primitive Fitting Network (SPFN), an end-to-end neural network that can robustly detect a varying n…
International Institute of Information Technology Bangalore（班加罗尔国际信息技术学院）
LiDAR point clouds provide rich geometric information, which is particularly useful for the analysis of complex scenes of urban regions. Finding structural and semantic differences between two different three-dimensional point clouds, say, of the same region but acquired at different time instances is an important problem. A comparison of point clouds involves computationally expensive registration and segmentation. We are interested in capturing the relative differences in the geometric uncertainty and semantic content of the point cloud without the registration process. Hence, we propose an …
GSTran: Joint Geometric and Semantic Coherence for Point Cloud Segmentation
Jiangxi University of Finance and Economics（江西财经大学）；Shenzhen University（深圳大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）；University of Technology Sydney（悉尼科技大学）
GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；Intelligent Science and Technology Academy of CASIC（中国航天科工集团智能科技研究院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Department of Automation, Tsinghua University（清华大学自动化系）
Analyzing the geometric and semantic properties of 3D point clouds through the deep networks is still challenging due to the irregularity and sparsity of samplings of their geometric structures. This paper presents a new method to define and compute convolution directly on 3D point clouds by the proposed annular convolution. This new convolution operator can better capture the local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processi…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Adobe（奥多比）；University of Cyprus（塞浦路斯大学）
We present PriFit, a semi-supervised approach for label-efficient learning of 3D point cloud segmentation networks. PriFit combines geometric primitive fitting with point-based representation learning. Its key idea is to learn point representations whose clustering reveals shape regions that can be approximated well by basic geometric primitives, such as cuboids and ellipsoids. The learned point representations can then be re-used in existing network architectures for 3D point cloud segmentation, and improves their performance in the few-shot setting. According to our experiments on the widely…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳市人工智能与机器人研究院）
In spite of the recent progresses on classifying 3D point cloud with deep CNNs, large geometric transformations like rotation and translation remain challenging problem and harm the final classification performance. To address this challenge, we propose Geometry Sharing Network (GS-Net) which effectively learns point descriptors with holistic context to enhance the robustness to geometric transformations. Compared with previous 3D point CNNs which perform convolution on nearby points, GS-Net can aggregate point features in a more global way. Specially, GS-Net consists of Geometry Similarity Co…
Technion - Israel Institute of Technology（以色列理工学院）
Reconstruction of geometric structures from images using supervised learning suffers from limited available amount of accurate data. One type of such data is accurate real-world RGB-D images. A major challenge in acquiring such ground truth data is the accurate alignment between RGB images and the point cloud measured by a depth scanner. To overcome this difficulty, we consider a differential optimization method that aligns a colored point cloud with a given color image through iterative geometric and color matching. In the proposed framework, the optimization minimizes the photometric differe…
Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）；West Virginia University（西弗吉尼亚大学）
Point cloud analysis is a fundamental task in 3D computer vision. Most previous works have conducted experiments on synthetic datasets with well-aligned data; while real-world point clouds are often not pre-aligned. How to achieve rotation invariance remains an open problem in point cloud analysis. To meet this challenge, we propose a novel approach toward achieving rotation-invariant (RI) representations by combining local geometry with global topology. In our local-global-representation (LGR)-Net, we have designed a two-branch network where one stream encodes local geometric RI features and …
GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model
Wangxuan Institute of Computer Technology, Peking University, Beijing, China（王轩计算机技术研究所，北京大学，北京，中国）；State Key Laboratory of CAD\&CG, Zhejiang University, Hangzhou, China（CAD\&CG国家重点实验室，浙江大学，杭州，中国）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China（人工智能与自动化学院，华中科技大学，武汉，中国）
Point cloud compression (PCC) is a key enabler for various 3-D applications, owing to the universality of the point cloud format. Ideally, 3D point clouds endeavor to depict object/scene surfaces that are continuous. Practically, as a set of discrete samples, point clouds are locally disconnected and sparsely distributed. This sparse nature is hindering the discovery of local correlation among points for compression. Motivated by an analysis with fractal dimension, we propose a heterogeneous approach with deep learning for lossy point cloud geometry compression. On top of a base layer compress…
