在本文中，我们探索了一种用于多视角图像三维占用预测的新型点表示方法，命名为“将占用表示为点集”（Occupancy as Set of Points）。现有的基于相机的方法倾向于利用稠密的体素表示来预测整个场景的占用情况，这使得它们难以聚焦于特殊区域或感知范围之外的区域。相比之下，我们提出使用兴趣点（Points of Interest, PoIs）来表示场景，并提出了OSP，一个用于基于点的三维占用预测的新型框架。由于点表示固有的灵活性，OSP在性能上相比现有方法表现出强大的竞争力，并在训练和推理适应性方面具有显著优势。它超越了传统的感知边界，并且可以与基于体素的方法无缝集成，从而显著提升其有效性。在Occ3D nuScenes占用基准上的实验表明，OSP具有强大的性能和灵活性。代码和模型可在\url{https://github.com/hustvl/osp}获取。
英文摘要：
In this paper, we explore a novel point representation for 3D occupancy prediction from multi-view images, which is named Occupancy as Set of Points. Existing camera-based methods tend to exploit dense volume-based representation to predict the occupancy of the whole scene, making it hard to focus on the special areas or areas out of the perception range. In comparison, we present the Points of Interest (PoIs) to represent the scene and propose OSP, a novel framework for point-based 3D occupancy prediction. Owing to the inherent flexibility of the point-based representation, OSP achieves strong performance compared with existing methods and excels in terms of training and inference adaptability. It extends beyond traditional perception boundaries and can be seamlessly integrated with volume-based methods to significantly enhance their effectiveness. Experiments on the Occ3D nuScenes occupancy benchmark show that OSP has strong performance and flexibility. Code and models are available at \url{https://github.com/hustvl/osp}.
Université Gustave Eiffel, IGN-ENSG, LASTIG（古斯塔夫·埃菲尔大学，法国国家地理与林业信息研究所-国家地理科学学院，地理信息与空间分析实验室）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，BIOGECO联合研究单位，波尔多大学）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多综合理工学院G&E研究组，波尔多蒙田大学）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from 3D point clouds captured from an aerial platform. Our model predicts rasterized occupancy maps for three vegetation strata: lower, medium, and higher strata. Our training scheme allows our network to only being supervized with values aggregated over cylindrical plots, which are easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision while simultaneously providing visual and interpretable predictions. We prov…
WildOcc: A Benchmark for Off-Road 3D Semantic Occupancy Prediction
Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能计算系统研究中心）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
AdaOcc: Adaptive 3D Occupancy Prediction for Embodied Tasks
Beihang University（北京航空航天大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；XYZ Embodied AI（XYZ具身智能公司）；Hebei University of Technology（河北工业大学）；ShanghaiTech University（上海科技大学）；University of Science and Technology Beijing（北京科技大学）
TGP: Two-modal occupancy prediction with 3D Gaussian and sparse points for 3D Environment Awareness
China Telecom Research Institute（中国电信研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
Semantic Occupancy Prediction with Dual Range-Voxel Representation
School of Software Engineering, South China University of Technology（华南理工大学软件学院）；State Key Laboratory of Optical Field Manipulation Science and Technology, Institute of Optics and Electronics, CAS（中国科学院光电技术研究所光学场操控科学与技术国家重点实验室）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook 人工智能研究院）
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent. We propose occupancy anticipation, where the agent uses its egocentric RGB-D observations to infer the occupancy state beyond the visible regions. In doing so, the agent builds its spatial awareness more rapidly, which facilitates efficient exploration and navigation in 3D environments. By exploiting context in both the egocentric views and top-down maps our model successfully anticipates a br…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-尚佩恩分校）
Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. Specifically, a set of 3D locations within the view-frustum of the camera are first projected independently onto the image and a corresponding feature is subsequently extracted for each 3D location. The feature of each 3D location is then used to classify independently whether the corresponding 3D point is inside or outside the observed object. This procedure leads to sub-optimal results because correlations between predictions for neighboring locations are only taken into account im…
South China University of Technology（华南理工大学）；Alibaba Group（阿里巴巴集团）；HKUST（香港科技大学）；Pazhou Lab（琶洲实验室）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）
Surface reconstruction from point clouds is a fundamental problem in the computer vision and graphics community. Recent state-of-the-arts solve this problem by individually optimizing each local implicit field during inference. Without considering the geometric relationships between local fields, they typically require accurate normals to avoid the sign conflict problem in overlapped regions of local fields, which severely limits their applicability to raw scans where surface normals could be unavailable. Although SAL breaks this limitation via sign-agnostic learning, further works still need …
Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
Shanghai Jiao Tong University（上海交通大学）；HAOMO.AI Technology Co., Ltd.（HAOMO.AI技术有限公司）；National University of Defense Technology（国防科技大学）；Beijing Institute of Technology（北京理工大学）
Predicting the possible future behaviors of vehicles that drive on shared roads is a crucial task for safe autonomous driving. Many existing approaches to this problem strive to distill all possible vehicle behaviors into a simplified set of high-level actions. However, these action categories do not suffice to describe the full range of maneuvers possible in the complex road networks we encounter in the real world. To combat this deficiency, we propose a new method that leverages the mapped road topology to reason over possible goals and predict the future spatial occupancy of dynamic road ac…
SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；School of Information Science and Technology（信息科学与技术学院）；Fudan University（复旦大学）；School of Artificial Intelligence（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）
OG-Gaussian: Occupancy Based Street Gaussians for Autonomous Driving
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）
GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sun Yat-sen University（中山大学）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Scan data of urban environments often include representations of dynamic objects, such as vehicles, pedestrians, and so forth. However, when it comes to constructing a 3D point cloud map with sequential accumulations of the scan data, the dynamic objects often leave unwanted traces in the map. These traces of dynamic objects act as obstacles and thus impede mobile vehicles from achieving good localization and navigation performances. To tackle the problem, this paper presents a novel static map building method called ERASOR, Egocentric RAtio of pSeudo Occupancy-based dynamic object Removal, wh…
Electrical Engineering and Computer Science York University（约克大学电气工程与计算机科学学院）
Implicit function based surface reconstruction has been studied for a long time to recover 3D shapes from point clouds sampled from surfaces. Recently, Signed Distance Functions (SDFs) and Occupany Functions are adopted in learning-based shape reconstruction methods as implicit 3D shape representation. This paper proposes a novel method for learning occupancy functions from sparse point clouds and achieves better performance on challenging surface reconstruction tasks. Unlike the previous methods, which predict point occupancy with fully-connected multi-layer networks, we adapt the point cloud…
LASTIG, Univ. Gustave Eiffel, IGN-ENSG（古斯塔夫·埃菲尔大学LASTIG实验室，法国国家地理与林业信息学院）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，波尔多大学生物地球化学与生态学联合研究中心）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多蒙田大学，波尔多国立高等电子、计算机、电信、数学与机械学院环境与地理空间工程研究团队）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from airborne 3D LiDAR point clouds. Our model predicts rasterized occupancy maps for three vegetation strata corresponding to lower, medium, and higher cover. Our weakly-supervised training scheme allows our network to only be supervised with vegetation occupancy values aggregated over cylindrical plots containing thousands of points. Such ground truth is easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision b…
