Transformers have become dominant in large-scale deep learning tasks across various domains, including text, 2D and 3D vision. However, the quadratic complexity of their attention mechanism limits their efficiency as the sequence length increases, particularly in high-resolution 3D data such as point clouds. Recently, state space models (SSMs) like Mamba have emerged as promising alternatives, offering linear complexity, scalability, and high performance in long-sequence tasks. The key challenge in the application of SSMs in this domain lies in reconciling the non-sequential structure of point clouds with the inherently directional (or bi-directional) order-dependent processing of recurrent models like Mamba. To achieve this, previous research proposed reorganizing point clouds along multiple directions or predetermined paths in 3D space, concatenating the results to produce a single 1D sequence capturing different views. In our work, we introduce a method to convert point clouds into 1D sequences that maintain 3D spatial structure with no need for data replication, allowing Mamba sequential processing to be applied effectively in an almost permutation-invariant manner. In contrast to other works, we found that our method does not require positional embeddings and allows for shorter sequence lengths while still achieving state-of-the-art results in ModelNet40 and ScanObjectNN datasets and surpassing Transformer-based models in both accuracy and efficiency.
Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
School of Computer Science, National Key Laboratory for Multimedia Information Processing, Peking University, China（计算机学院、多媒体信息处理国家重点实验室、北京大学）；School of Physics, Peking University, China（物理学院、北京大学）
Duke University（杜克大学）；Geometric Data Analytics, Inc.（几何数据分析公司）；University of Wisconsin-Stout（威斯康星大学斯托特分校）；Air Force Research Laboratory（空军研究实验室）
In this work, we address the problem of cross-modal comparison of aerial data streams. A variety of simulated automobile trajectories are sensed using two different modalities: full-motion video, and radio-frequency (RF) signals received by detectors at various locations. The information represented by the two modalities is compared using self-similarity matrices (SSMs) corresponding to time-ordered point clouds in feature spaces of each of these data sources; we note that these feature spaces can be of entirely different scale and dimensionality. Several metrics for comparing SSMs are explore…
UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（一般人工智能国家重点实验室，北京大学深圳研究生院）；The Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学 Urbana-Champaign 院，浙江大学）；State University of New York at Buffalo（纽约州立大学布法罗分校）
PointDGMamba: Domain Generalization of Point Cloud Classification via Generalized State Space Model
Shanghai Jiao Tong University（上海交通大学）；Nanjing University（南京大学）；Skywork AI；The University of Western Australia（西澳大学）；Nanyang Technological University（南洋理工大学）
Duke University（杜克大学）；Geometric Data Analytics, Inc.（几何数据分析公司）
In this work, we address fusion of heterogeneous sensor data using wavelet-based summaries of fused self-similarity information from each sensor. The technique we develop is quite general, does not require domain specific knowledge or physical models, and requires no training. Nonetheless, it can perform surprisingly well at the general task of differentiating classes of time-ordered behavior sequences which are sensed by more than one modality. As a demonstration of our capabilities in the audio to video context, we focus on the differentiation of speech sequences. Data from two or more modal…
Institute for Infocomm Research（资讯通信研究院）；A*STAR（新加坡科技研究局）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Electronic Science and Technology of China（电子科技大学）
Data augmentation is an important technique to reduce overfitting and improve learning performance, but existing works on data augmentation for 3D point cloud data are based on heuristics. In this work, we instead propose to automatically learn a data augmentation strategy using bilevel optimization. An augmentor is designed in a similar fashion to a conditional generator and is optimized by minimizing a base model's loss on a validation set when the augmented input is used for training the model. This formulation provides a more principled way to learn data augmentation on 3D point clouds. We…
As 3D point clouds become the representation of choice for multiple vision and graphics applications, the ability to synthesize or reconstruct high-resolution, high-fidelity point clouds becomes crucial. Despite the recent success of deep learning models in discriminative tasks of point clouds, generating point clouds remains challenging. This paper proposes a principled probabilistic framework to generate 3D point clouds by modeling them as a distribution of distributions. Specifically, we learn a two-level hierarchy of distributions where the first level is the distribution of shapes and the…
In this work, we explore the problem of aligning two time-ordered point clouds which are spatially transformed and re-parameterized versions of each other. This has a diverse array of applications such as cross modal time series synchronization (e.g. MOCAP to video) and alignment of discretized curves in images. Most other works that address this problem attempt to jointly uncover a spatial alignment and correspondences between the two point clouds, or to derive local invariants to spatial transformations such as curvature before computing correspondences. By contrast, we sidestep spatial alig…
ZigzagPointMamba: Spatial-Semantic Mamba for Point Cloud Understanding
Key Laboratory of Signal Detection and Processing, Xinjiang University（信号检测与处理重点实验室，新疆大学）；Joint International Research Laboratory of Silk Road Multilingual Cognitive Computing, Xinjiang University（丝绸之路多语种认知计算联合国际实验室，新疆大学）；National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）
TRACE: Artifact-Robust Statistical Shape Modeling from Imperfect Surface Scans - A Case Study in Craniosynostosis 3D Photography
Scientific Computing and Imaging Institute, University of Utah（犹他大学科学计算与成像研究所）；Kahlert School of Computing, University of Utah（犹他大学卡勒特计算学院）；Division of Pediatric Plastic Surgery, UPMC Children’s Hospital of Pittsburgh（匹兹堡大学医学中心儿童医院小儿整形外科）
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Real-scanned point clouds are often incomplete due to viewpoint, occlusion, and noise. Existing point cloud completion methods tend to generate global shape skeletons and hence lack fine local details. Furthermore, they mostly learn a deterministic partial-to-complete mapping, but overlook structural relations in man-made objects. To tackle these challenges, this paper proposes a variational framework, Variational Relational point Completion network (VRCNet) with two appealing properties: 1) Probabilistic Modeling. In particular, we propose a dual-path architecture to enable principled probabi…
Latent Mamba Operator for Partial Differential Equations
Department of Electrical Communication Engineering, Indian Institute of Science, Bangalore, India（电子通信工程系，印度科学研究院，班加罗尔，印度）；Yardi School of AI, Indian Institute of Technology, New Delhi, India（AI学院，印度理工学院，新德里，印度）；Department of Civil Engineering, Indian Institute of Technology, New Delhi, India（土木工程系，印度理工学院，新德里，印度）；Department of Computer Science and Automation, Indian Institute of Science, Bangalore, India（计算机科学与自动化系，印度科学研究院，班加罗尔，印度）
针对神经算子求解偏微分方程时面临的高维扩展性差、计算成本高及难以捕捉长程依赖等问题，提出 Latent Mamba Operator (LaMO)，结合潜空间状态空间模型与核积分公式，并在多种 PDE 基准上取得 SOTA 性能。
Point-clouds are a popular choice for vision and graphics tasks due to their accurate shape description and direct acquisition from range-scanners. This demands the ability to synthesize and reconstruct high-quality point-clouds. Current deep generative models for 3D data generally work on simplified representations (e.g., voxelized objects) and cannot deal with the inherent redundancy and irregularity in point-clouds. A few recent efforts on 3D point-cloud generation offer limited resolution and their complexity grows with the increase in output resolution. In this paper, we develop a princip…
Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）；West Virginia University（西弗吉尼亚大学）
Point cloud analysis is a fundamental task in 3D computer vision. Most previous works have conducted experiments on synthetic datasets with well-aligned data; while real-world point clouds are often not pre-aligned. How to achieve rotation invariance remains an open problem in point cloud analysis. To meet this challenge, we propose a novel approach toward achieving rotation-invariant (RI) representations by combining local geometry with global topology. In our local-global-representation (LGR)-Net, we have designed a two-branch network where one stream encodes local geometric RI features and …
Centre Hospitalier de Luxembourg（卢森堡医院）；University of Luxembourg（卢森堡大学）；Zuse Institute Berlin（柏林楚泽研究所）；shapes GmbH（1000shapes有限责任公司）；Trier University of Applied Sciences（特里尔应用技术大学）
The reconstruction of an object's shape or surface from a set of 3D points plays an important role in medical image analysis, e.g. in anatomy reconstruction from tomographic measurements or in the process of aligning intra-operative navigation and preoperative planning data. In such scenarios, one usually has to deal with sparse data, which significantly aggravates the problem of reconstruction. However, medical applications often provide contextual information about the 3D point data that allow to incorporate prior knowledge about the shape that is to be reconstructed. To this end, we propose…
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Point cloud segmentation is a fundamental task in 3D vision that serves a wide range of applications. Although great progresses have been made these years, its practical usability is still limited by the availability of training data. Existing approaches cannot make full use of multiple datasets on hand due to the label mismatch among different datasets. In this paper, we propose a principled approach that supports learning from heterogeneous datasets with different label sets. Our idea is to utilize a pre-trained language model to embed discrete labels to a continuous latent space with the he…
In this paper, we unify popular non-rigid registration methods for point sets and surfaces under our general framework, GiNGR. GiNGR builds upon Gaussian Process Morphable Models (GPMM) and hence separates modeling the deformation prior from model adaptation for registration. In addition, it provides explainable hyperparameters, multi-resolution registration, trivial inclusion of expert annotation, and the ability to use and combine analytical and statistical deformation priors. But more importantly, the reformulation allows for a direct comparison of registration methods. Instead of using a g…
Developing neural models that accurately understand objects in 3D point clouds is essential for the success of robotics and autonomous driving. However, arguably due to the higher-dimensional nature of the data (as compared to images), existing neural architectures exhibit a large variety in their designs, including but not limited to the views considered, the format of the neural features, and the neural operations used. Lack of a unified framework and interpretation makes it hard to put these designs in perspective, as well as systematically explore new ones. In this paper, we begin by propo…
We present SyNoRiM, a novel way to jointly register multiple non-rigid shapes by synchronizing the maps relating learned functions defined on the point clouds. Even though the ability to process non-rigid shapes is critical in various applications ranging from computer animation to 3D digitization, the literature still lacks a robust and flexible framework to match and align a collection of real, noisy scans observed under occlusions. Given a set of such point clouds, our method first computes the pairwise correspondences parameterized via functional maps. We simultaneously learn potentially n…
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Learning to reconstruct 3D garments is important for dressing 3D human bodies of different shapes in different poses. Previous works typically rely on 2D images as input, which however suffer from the scale and pose ambiguities. To circumvent the problems caused by 2D images, we propose a principled framework, Garment4D, that uses 3D point cloud sequences of dressed humans for garment reconstruction. Garment4D has three dedicated steps: sequential garments registration, canonical garment estimation, and posed garment reconstruction. The main challenges are two-fold: 1) effective 3D feature lea…
OverlapMamba: Novel Shift State Space Model for LiDAR-based Place Recognition
School of Data Science and Engineering, Xingzhi College, South China Normal University（华南师范大学行知学院数据科学与工程学院）；School of Electronic and Information Engineering, South China Normal University（华南师范大学电子与信息工程学院）；Hong Kong University of Science and Technology（香港科技大学）；College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学学院）；College of Electronics & Information Engineering, Shanghai Research Institu（上海电子信息工程研究院）
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；McGill University（麦吉尔大学）；Intel Labs China, Peking University（北京大学英特尔中国实验室）
Semantic understanding of 3D point clouds is important for various robotics applications. Given that point-wise semantic annotation is expensive, in this paper, we address the challenge of learning models with extremely sparse labels. The core problem is how to leverage numerous unlabeled points. To this end, we propose a self-supervised 3D representation learning framework named viewpoint bottleneck. It optimizes a mutual-information based objective, which is applied on point clouds under different viewpoints. A principled analysis shows that viewpoint bottleneck leads to an elegant surrogate…
GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）
