Learning Geometry-Disentangled Representation for Complementary Understanding of 3D Object Point Cloud
Guangdong-Hong Kong-Macao Joint Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（广东省粤港澳人机智能协同系统联合实验室，中国科学院深圳先进技术研究院）；The University of Hong Kong（香港大学）；University of Chinese Academy of Sciences（中国科学院大学）
In 2D image processing, some attempts decompose images into high and low frequency components for describing edge and smooth parts respectively. Similarly, the contour and flat area of 3D objects, such as the boundary and seat area of a chair, describe different but also complementary geometries. However, such investigation is lost in previous deep networks that understand point clouds by directly treating all points or local patches equally. To solve this problem, we propose Geometry-Disentangled Attention Network (GDANet). GDANet introduces Geometry-Disentangle Module to dynamically disentangle point clouds into the contour and flat part of 3D objects, respectively denoted by sharp and gentle variation components. Then GDANet exploits Sharp-Gentle Complementary Attention Module that regards the features from sharp and gentle variation components as two holistic representations, and pays different attentions to them while fusing them respectively with original point cloud features. In this way, our method captures and refines the holistic and complementary 3D geometric semantics from two distinct disentangled components to supplement the local information. Extensive experiments on 3D object classification and segmentation benchmarks demonstrate that GDANet achieves the state-of-the-arts with fewer parameters. Code is released on https://github.com/mutianxu/GDANet.
Point clouds produced by 3D scanning are often sparse, non-uniform, and noisy. Recent upsampling approaches aim to generate a dense point set, while achieving both distribution uniformity and proximity-to-surface, and possibly amending small holes, all in a single network. After revisiting the task, we propose to disentangle the task based on its multi-objective nature and formulate two cascaded sub-networks, a dense generator and a spatial refiner. The dense generator infers a coarse but dense output that roughly describes the underlying surface, while the spatial refiner further fine-tunes t…
Australian National University（澳大利亚国立大学）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）
This paper tackles the problem of parts-aware point cloud generation. Unlike existing works which require the point cloud to be segmented into parts a priori, our parts-aware editing and generation are performed in an unsupervised manner. We achieve this with a simple modification of the Variational Auto-Encoder which yields a joint model of the point cloud itself along with a schematic representation of it as a combination of shape primitives. In particular, we introduce a latent representation of the point cloud which can be decomposed into a disentangled representation for each part of the …
Peking University（北京大学）；Chinese University of Hong Kong（香港中文大学）
Semantic scene reconstruction from point cloud is an essential and challenging task for 3D scene understanding. This task requires not only to recognize each instance in the scene, but also to recover their geometries based on the partial observed point cloud. Existing methods usually attempt to directly predict occupancy values of the complete object based on incomplete point cloud proposals from a detection-based backbone. However, this framework always fails to reconstruct high fidelity mesh due to the obstruction of various detected false positive object proposals and the ambiguity of inco…
UC San Diego（加州大学圣迭戈分校）；Johns Hopkins University（约翰斯·霍普金斯大学）
Recent work has made significant progress on using implicit functions, as a continuous representation for 3D rigid object shape reconstruction. However, much less effort has been devoted to modeling general articulated objects. Compared to rigid objects, articulated objects have higher degrees of freedom, which makes it hard to generalize to unseen shapes. To deal with the large shape variance, we introduce Articulated Signed Distance Functions (A-SDF) to represent articulated shapes with a disentangled latent space, where we have separate codes for encoding shape and articulation. We assume n…
Interpretable Single-View 3D Gaussian Splatting using Unsupervised Hierarchical Disentangled Representation Learning
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Hong Kong Polytechnic University（香港理工大学）
We present MRGAN, a multi-rooted adversarial network which generates part-disentangled 3D point-cloud shapes without part-based shape supervision. The network fuses multiple branches of tree-structured graph convolution layers which produce point clouds, with learnable constant inputs at the tree roots. Each branch learns to grow a different shape part, offering control over the shape generation at the part level. Our network encourages disentangled generation of semantic parts via two key ingredients: a root-mixing training strategy which helps decorrelate the different branches to facilitate…
Tsinghua University（清华大学）；BNRist（北京国家信息科学与技术研究中心）；University of Washington（华盛顿大学）
While category-level 9DoF object pose estimation has emerged recently, previous correspondence-based or direct regression methods are both limited in accuracy due to the huge intra-category variances in object shape and color, etc. Orthogonal to them, this work presents a category-level object pose and size refiner CATRE, which is able to iteratively enhance pose estimate from point clouds to produce accurate results. Given an initial pose estimate, CATRE predicts a relative transformation between the initial pose and ground truth by means of aligning the partially observed point cloud and an …
Easi3R: Estimating Disentangled Motion from DUSt3R Without Training
Zhejiang University（浙江大学）；Westlake University（西湖大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Tübingen, Tübingen AI Center（图宾根大学，图宾根AI中心）
Korea University（高丽大学）；Meta Reality Labs（Meta现实实验室）
Transformers have shown superior performance on various computer vision tasks with their capabilities to capture long-range dependencies. Despite the success, it is challenging to directly apply Transformers on point clouds due to their quadratic cost in the number of points. In this paper, we present a Self-Positioning point-based Transformer (SPoTr), which is designed to capture both local and global shape contexts with reduced complexity. Specifically, this architecture consists of local self-attention and self-positioning point-based global cross-attention. The self-positioning points, ada…
PAGE-4D: Disentangled pose and geometry estimation for vggt-4d perception
Harvard AI and Robotics Lab, Harvard University（哈佛人工智能与机器人实验室，哈佛大学）；Media Lab and Electrical Engineering and Computer Science, Massachusetts Institute of Technology（媒体实验室和电气工程与计算机科学，麻省理工学院）；Department of Computing, Imperial College London（计算系，帝国理工学院）；Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University（哈佛大学自然与人工智能研究学院）
Point Cloud Mixture-of-Domain-Experts Model for 3D Self-supervised Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Institute of Perceptual Intelligence, Pengcheng Laboratory（感知智能研究院，鹏城实验室）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
Camera sensors are increasingly being combined with machine learning to perform various tasks such as intelligent surveillance. Due to its computational complexity, most of these machine learning algorithms are offloaded to the cloud for processing. However, users are increasingly concerned about privacy issues such as function creep and malicious usage by third-party cloud providers. To alleviate this, we propose an edge-based filtering stage that removes privacy-sensitive attributes before the sensor data are transmitted to the cloud. We use state-of-the-art image manipulation techniques tha…
This paper investigates an open research task of reconstructing and generating 3D point clouds. Most existing works of 3D generative models directly take the Gaussian prior as input for the decoder to generate 3D point clouds, which fail to learn disentangled latent codes, leading noisy interpolated results. Most of the GAN-based models fail to discriminate the local geometries, resulting in the point clouds generated not evenly distributed at the object surface, hence degrading the point cloud generation quality. Moreover, prevailing methods adopt computation-intensive frameworks, such as flo…
National Engineering Laboratory for Video Technology, Peking University（北京大学视频技术国家工程实验室）
Place recognition is one of the hot research fields in automation technology and is still an open issue, Camera and Lidar are two mainstream sensors used in this task, Camera-based methods are easily affected by illumination and season changes, LIDAR cannot get the rich data as the image could , In this paper, we propose the PIC-Net (Point cloud and Image Collaboration Network), which use attention mechanism to fuse the features of image and point cloud, and mine the complementary information between the two. Furthermore, in order to improve the recognition performance at night, we transform t…
RI-MAE: Rotation-Invariant Masked AutoEncoders for Self-Supervised Point Cloud Representation Learning
South China University of Technology（华南理工大学）；The University of Adelaide（阿德莱德大学）；The University of Western Australia（西澳大学）；The University of Sydney（悉尼大学）
Beihang University（北京航空航天大学）；The Chinese University of Hong Kong（香港中文大学）
3D human mesh recovery from point clouds is essential for various tasks, including AR/VR and human behavior understanding. Previous works in this field either require high-quality 3D human scans or sequential point clouds, which cannot be easily applied to low-quality 3D scans captured by consumer-level depth sensors. In this paper, we make the first attempt to reconstruct reliable 3D human shapes from single-frame partial point clouds.To achieve this, we propose an end-to-end learnable method, named VoteHMR. The core of VoteHMR is a novel occlusion-aware voting network that can first reliably…
Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）；West Virginia University（西弗吉尼亚大学）
Point cloud analysis is a fundamental task in 3D computer vision. Most previous works have conducted experiments on synthetic datasets with well-aligned data; while real-world point clouds are often not pre-aligned. How to achieve rotation invariance remains an open problem in point cloud analysis. To meet this challenge, we propose a novel approach toward achieving rotation-invariant (RI) representations by combining local geometry with global topology. In our local-global-representation (LGR)-Net, we have designed a two-branch network where one stream encodes local geometric RI features and …
Peking University（北京大学）；Heisenberg Robotics（海森堡机器人公司）；University of Pennsylvania（宾夕法尼亚大学）
Performances on standard 3D point cloud benchmarks have plateaued, resulting in oversized models and complex network design to make a fractional improvement. We present an alternative to enhance existing deep neural networks without any redesigning or extra parameters, termed as Spatial-Neighbor Adapter (SN-Adapter). Building on any trained 3D network, we utilize its learned encoding capability to extract features of the training dataset and summarize them as prototypical spatial knowledge. For a test point cloud, the SN-Adapter retrieves k nearest neighbors (k-NN) from the pre-constructed spa…
In this paper, we present a novel differential morph detection framework, utilizing landmark and appearance disentanglement. In our framework, the face image is represented in the embedding domain using two disentangled but complementary representations. The network is trained by triplets of face images, in which the intermediate image inherits the landmarks from one image and the appearance from the other image. This initially trained network is further trained for each dataset using contrastive representations. We demonstrate that, by employing appearance and landmark disentanglement, the pr…
Masked Clustering Prediction for Unsupervised Point Cloud Pre-training
University of Pisa（比萨大学）；University of Trento（特伦托大学）；Shanghai Jiao Tong University（上海交通大学）；Peking University（北京大学）；Fondazione Bruno Kessler（布鲁诺·凯索尔基金会）
Generative models that learn disentangled representations for different factors of variation in an image can be very useful for targeted data augmentation. By sampling from the disentangled latent subspace of interest, we can efficiently generate new data necessary for a particular task. Learning disentangled representations is a challenging problem, especially when certain factors of variation are difficult to label. In this paper, we introduce a novel architecture that disentangles the latent space into two complementary subspaces by using only weak supervision in form of pairwise similarity…
Point clouds provide a compact and efficient representation of 3D shapes. While deep neural networks have achieved impressive results on point cloud learning tasks, they require massive amounts of manually labeled data, which can be costly and time-consuming to collect. In this paper, we leverage 3D self-supervision for learning downstream tasks on point clouds with fewer labels. A point cloud can be rotated in infinitely many ways, which provides a rich label-free source for self-supervision. We consider the auxiliary task of predicting rotations that in turn leads to useful features for othe…
The Chinese University of Hong Kong（香港中文大学）；SenseTime Group Limited（商汤集团有限公司）；Peking University（北京大学）
This paper proposes learning disentangled but complementary face features with minimal supervision by face identification. Specifically, we construct an identity Distilling and Dispelling Autoencoder (D2AE) framework that adversarially learns the identity-distilled features for identity verification and the identity-dispelled features to fool the verification system. Thanks to the design of two-stream cues, the learned disentangled features represent not only the identity or attribute but the complete input image. Comprehensive evaluations further demonstrate that the proposed features not onl…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen University（深圳大学）；Renmin University of China（中国人民大学）
Fusing medical images and the corresponding 3D shape representation can provide complementary information and microstructure details to improve the operational performance and accuracy in brain surgery. However, compared to the substantial image data, it is almost impossible to obtain the intraoperative 3D shape information by using physical methods such as sensor scanning, especially in minimally invasive surgery and robot-guided surgery. In this paper, a general generative adversarial network (GAN) architecture based on graph convolutional networks is proposed to reconstruct the 3D point clo…
DiCo: Disentangled Concept Representation for Text-to-image Person Re-identification
organization= Department of Imaging Science, Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University , addressline= , city= Seoul , postcode= 06974 , state= , country= South Korea；organization= Department of Metaverse Convergence, Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University , addressline= , city= Seoul , postcode= 06974 , state= , country= South Korea
DrFER: Learning Disentangled Representations for 3D Facial Expression Recognition
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding
School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；State Key Laboratory of Intelligent Geotechnics and Tunnelling (FSDI)（智能岩土与隧道工程国家重点实验室）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））；School of Information and Communication Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）
Dual Consistent Constraint via Disentangled Consistency and Complementarity for Multi-view Clustering
College of Data Science and Application（数据科学与应用学院）；Inner Mongolia Autonomous Region Engineering and Technology Research Center of Big Data-Based Software Service（内蒙古自治区大数据软件服务工程技术研究中心）；Inner Mongolia Key Laboratory of Beijiang Cyberspace Security（内蒙古北疆网络空间安全重点实验室）
We present a simple neural rendering architecture that helps variational autoencoders (VAEs) learn disentangled representations. Instead of the deconvolutional network typically used in the decoder of VAEs, we tile (broadcast) the latent vector across space, concatenate fixed X- and Y-"coordinate" channels, and apply a fully convolutional network with 1x1 stride. This provides an architectural prior for dissociating positional from non-positional features in the latent distribution of VAEs, yet without providing any explicit supervision to this effect. We show that this architecture, which we …
Adaptive Spectrum-Aware Feature Disentangled Network for Small Object Detection
Shenzhen Campus of Sun Yat-sen University（孙中山大学深圳校区）；Beijing University of Posts and Telecommunications（北京邮电大学）；Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院）；Renmin University of China（中国人民大学）；CITIC Bank（中国 CITIC 银行）
Learning Disentangled Representations for Generalized Multi-view Clustering
AI Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）人工智能方向）；School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Computer, National University of Defense Technology（国防科技大学计算机学院）；Medical Big Data Research Center, Medical Engineering Laboratory of Chinese PLA General Hospital（中国人民解放军总医院医学大数据研究中心，医学工程实验室）；School of Computing and Information Technology, University of Wollongong（沃林根大学计算与信息学院）
Multi-view clustering, a long-standing and important research problem, focuses on mining complementary information from diverse views. However, existing works often fuse multiple views' representations or handle clustering in a common feature space, which may result in their entanglement especially for visual representations. To address this issue, we present a novel VAE-based multi-view clustering framework (Multi-VAE) by learning disentangled visual representations. Concretely, we define a view-common variable and multiple view-peculiar variables in the generative model. The prior of view-co…
Point cloud semantic segmentation from projected views, such as range-view (RV) and bird's-eye-view (BEV), has been intensively investigated. Different views capture different information of point clouds and thus are complementary to each other. However, recent projection-based methods for point cloud semantic segmentation usually utilize a vanilla late fusion strategy for the predictions of different views, failing to explore the complementary information from a geometric perspective during the representation learning. In this paper, we introduce a geometric flow network (GFNet) to explore th…
