Autonomous driving requires robust perception across diverse environmental conditions, yet 3D semantic occupancy prediction remains challenging under adverse weather and lighting. In this work, we present the first study combining 4D radar and camera data for 3D semantic occupancy prediction. Our fusion leverages the complementary strengths of both modalities: 4D radar provides reliable range, velocity, and angle measurements in challenging conditions, while cameras contribute rich semantic and texture information. We further show that integrating depth cues from camera pixels enables lifting 2D images to 3D, improving scene reconstruction accuracy. Additionally, we introduce a fully automatically labeled dataset for training semantic occupancy models, substantially reducing reliance on costly manual annotation. Experiments demonstrate the robustness of 4D radar across diverse scenarios, highlighting its potential to advance autonomous vehicle perception.
DSOcc: Leveraging Depth Awareness and Semantic Aid to Boost Camera-Based 3D Semantic Occupancy Prediction
S-Lab & College of Computing and Data Science, Nanyang Technological University（S实验室及计算与数据科学学院，南洋理工大学）；MMLab, The Chinese University of Hong Kong（M实验室，香港中文大学）；State Key Laboratory of Fluid Power & Mechatronic Systems, Zhejiang University（流体动力与机电系统国家重点实验室，浙江大学）；SenseTime Research, Hong Kong（商汤研究，香港）
Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications
Shanghai Jiao Tong University（上海交通大学）；HAOMO.AI Technology Co., Ltd.（HAOMO.AI技术有限公司）；National University of Defense Technology（国防科技大学）；Beijing Institute of Technology（北京理工大学）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook 人工智能研究院）
State-of-the-art navigation methods leverage a spatial memory to generalize to new environments, but their occupancy maps are limited to capturing the geometric structures directly observed by the agent. We propose occupancy anticipation, where the agent uses its egocentric RGB-D observations to infer the occupancy state beyond the visible regions. In doing so, the agent builds its spatial awareness more rapidly, which facilitates efficient exploration and navigation in 3D environments. By exploiting context in both the egocentric views and top-down maps our model successfully anticipates a br…
Université Gustave Eiffel, IGN-ENSG, LASTIG（古斯塔夫·埃菲尔大学，法国国家地理与林业信息研究所-国家地理科学学院，地理信息与空间分析实验室）；INRAE, UMR 1202 BIOGECO, Université de Bordeaux（法国国家农业食品与环境研究院，BIOGECO联合研究单位，波尔多大学）；EA G&E Bordeaux INP, Université Bordeaux Montaigne（波尔多综合理工学院G&E研究组，波尔多蒙田大学）
We propose a new deep learning-based method for estimating the occupancy of vegetation strata from 3D point clouds captured from an aerial platform. Our model predicts rasterized occupancy maps for three vegetation strata: lower, medium, and higher strata. Our training scheme allows our network to only being supervized with values aggregated over cylindrical plots, which are easier to produce than pixel-wise or point-wise annotations. Our method outperforms handcrafted and deep learning baselines in terms of precision while simultaneously providing visual and interpretable predictions. We prov…
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of CS, Peking University（北京大学计算机学院）；Xiaomi Car（小米汽车）；Nanjing University（南京大学）；Nanyang Technological University（南洋理工大学）；CUHK MMLAB（香港中文大学多媒体实验室）
GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory
The Hong Kong Polytechnic University（香港理工大学）；Eastern Institute of Technology（东部理工学院）；Shanghai Jiao Tong University（上海交通大学）；Wuhan University（武汉大学）；The University of Hong Kong（香港大学）；Georgia Institute of Technology（佐治亚理工学院）
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
Predicting future behaviors of road agents is a key task in autonomous driving. While existing models have demonstrated great success in predicting marginal agent future behaviors, it remains a challenge to efficiently predict consistent joint behaviors of multiple agents. Recently, the occupancy flow fields representation was proposed to represent joint future states of road agents through a combination of occupancy grid and flow, which supports efficient and consistent joint predictions. In this work, we propose a novel occupancy flow fields predictor to produce accurate occupancy and flow p…
University of Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Inria（法国国家信息与自动化研究所）；Univ Lyon（里昂大学）；INSA Lyon（里昂国立应用科学学院）
Reliably predicting future occupancy of highly dynamic urban environments is an important precursor for safe autonomous navigation. Common challenges in the prediction include forecasting the relative position of other vehicles, modelling the dynamics of vehicles subjected to different traffic conditions, and vanishing surrounding objects. To tackle these challenges, we propose a spatio-temporal prediction network pipeline that takes the past information from the environment and semantic labels separately for generating future occupancy predictions. Compared to the current SOTA, our approach p…
We introduce a motion forecasting (behavior prediction) method that meets the latency requirements for autonomous driving in dense urban environments without sacrificing accuracy. A whole-scene sparse input representation allows StopNet to scale to predicting trajectories for hundreds of road agents with reliable latency. In addition to predicting trajectories, our scene encoder lends itself to predicting whole-scene probabilistic occupancy grids, a complementary output representation suitable for busy urban environments. Occupancy grids allow the AV to reason collectively about the behavior o…
WildOcc: A Benchmark for Off-Road 3D Semantic Occupancy Prediction
Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所智能计算系统研究中心）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Wangxuan Institute of Computer Technology, Peking University（北京大学计算机技术研究院）；Chongqing Changan Automobile Co., Ltd（重庆长安汽车有限公司）；University of California, Merced（加州大学默塞德分校）
OccTENS: 3D Occupancy World Model via Temporal Next-Scale Prediction
The Hong Kong University of Science and Technology（香港科学与技术大学）；University of Chinese Academy of Sciences（中国科学院大学）；Horizon Robotics（地平线机器人）；Nanjing University（南京大学）
