3D semantic occupancy prediction aims to forecast detailed geometric and semantic information of the surrounding environment for autonomous vehicles (AVs) using onboard surround-view cameras. Existing methods primarily focus on intricate inner structure module designs to improve model performance, such as efficient feature sampling and aggregation processes or intermediate feature representation formats. In this paper, we explore multitask learning by introducing an additional 3D supervision signal by incorporating an additional 3D object detection auxiliary branch. This extra 3D supervision signal enhances the model's overall performance by strengthening the capability of the intermediate features to capture small dynamic objects in the scene, and these small dynamic objects often include vulnerable road users, i.e. bicycles, motorcycles, and pedestrians, whose detection is crucial for ensuring driving safety in autonomous vehicles. Extensive experiments conducted on the nuScenes datasets, including challenging rainy and nighttime scenarios, showcase that our approach attains state-of-the-art results, achieving an IoU score of 31.73% and a mIoU score of 20.91% and excels at detecting vulnerable road users (VRU). The code will be made available at:https://github.com/DanielMing123/Inverse++
Prediction of dynamic environment is crucial to safe navigation of an autonomous vehicle. Urban traffic scenes are particularly challenging to forecast due to complex interactions between various dynamic agents, such as vehicles and vulnerable road users. Previous approaches have used egocentric occupancy grid maps to represent and predict dynamic environments. However, these predictions suffer from blurriness, loss of scene structure at turns, and vanishing of agents over longer prediction horizon. In this work, we propose a novel framework to make long-term predictions by representing the tr…
OC-SOP: Enhancing Vision-Based 3D Semantic Occupancy Prediction by Object-Centric Awareness
Autonomous Intelligent Systems group, Computer Science Institute VI – Intelligent Systems and Robotics – and the Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany（自主智能系统组，计算机科学研究所VI——智能系统与机器人——以及机器人中心和拉玛尔人工智能与机器学习研究所，波恩大学，德国）
Motion planning for safe autonomous driving requires learning how the environment around an ego-vehicle evolves with time. Ego-centric perception of driveable regions in a scene not only changes with the motion of actors in the environment, but also with the movement of the ego-vehicle itself. Self-supervised representations proposed for large-scale planning, such as ego-centric freespace, confound these two motions, making the representation difficult to use for downstream motion planners. In this paper, we use geometric occupancy as a natural alternative to view-dependent representations suc…
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of CS, Peking University（北京大学计算机学院）；Xiaomi Car（小米汽车）；Nanjing University（南京大学）；Nanyang Technological University（南洋理工大学）；CUHK MMLAB（香港中文大学多媒体实验室）
UniScene: Unified Occupancy-centric Driving Scene Generation
Shanghai Jiao Tong University（上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（东方理工大学宁波数字孪生研究院）；Tsinghua University（清华大学）；MEGVII Technology（旷视科技）；Mach Drive（马赫驱动）；Fudan University（复旦大学）；University of Hong Kong（香港大学）
SA-Occ: Satellite-Assisted 3D Occupancy Prediction in Real World
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
SuperOcc: Toward Cohesive Temporal Modeling for Superquadric-based 3D Occupancy Prediction
School of Control Science and Engineering, Dalian University of Technology, Dalian 116024, China（大连理工大学控制科学与工程学院，大连 116024，中国）；Dalian Rail Transmit Intelligent Control and Intelligent Operation Technology Innovation Center, Dalian 116024, China（大连铁路传输智能控制与智能运维技术创新中心，大连 116024，中国）；Dalian Seasky Automation Co., Ltd, Dalian 116024, China（大连海 sky 自动化有限公司，大连 116024，中国）
Manually specifying features that capture the diversity in traffic environments is impractical. Consequently, learning-based agents cannot realize their full potential as neural motion planners for autonomous vehicles. Instead, this work proposes to learn which features are task-relevant. Given its immediate relevance to motion planning, our proposed architecture encodes the probabilistic occupancy map as a proxy for obtaining pre-trained state representations. By leveraging a map-aware graph formulation of the environment, our agent-centric encoder generalizes to arbitrary road networks and t…
The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Human-centric perceptions include a variety of vision tasks, which have widespread industrial applications, including surveillance, autonomous driving, and the metaverse. It is desirable to have a general pretrain model for versatile human-centric downstream tasks. This paper forges ahead along this path from the aspects of both benchmark and pretraining methods. Specifically, we propose a \textbf{HumanBench} based on existing datasets to comprehensively evaluate on the common ground the generalization abilities of different pretraining methods on 19 datasets from 6 diverse downstream tasks, i…
GPOcc++: Unified Sparse Gaussian Occupancy Prediction with Visual Geometry Priors
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Sun Yat-sen University（中山大学）；The Hong Kong Polytechnic University（香港理工大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving
State Key Laboratory of Extreme Photonics and Instrumentation and the National Engineering Research Center of Optical Instrumentation（极端光物理与仪器国家重点实验室和光学仪器国家工程研究中心）；National Engineering Research Center of Optical Instrumentation（光学仪器国家工程研究中心）；College of Computer Science and Technology（计算机科学与技术学院）；Institute for Anthropomatics and Robotics（人机一体化机器人研究所）；Department of Engineering（工程系）
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）；PhiGent Robotics（鉴智机器人）
Modern methods for vision-centric autonomous driving perception widely adopt the bird's-eye-view (BEV) representation to describe a 3D scene. Despite its better efficiency than voxel representation, it has difficulty describing the fine-grained 3D structure of a scene with a single plane. To address this, we propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. To lift image features to the 3D TPV space, we further propose a transformer-based …
GEOcc: Geometrically Enhanced 3D Occupancy Network with Implicit-Explicit Depth Fusion and Contextual Self-Supervision
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；School of Electronics, Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；School of Transportation and Traffic Engineering, Central South University（中南大学交通运输工程学院）
COME: Adding Scene-Centric Forecasting Control to Occupancy World Model
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；State Key Laboratory of Intelligent Green Vehicle and Mobility（智能绿色车辆与移动系统国家重点实验室）；Kargobot Inc（Kargobot公司）；Nankai University（南开大学）
Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology（东部技术研究院）；School of Electronic Information and Electrical Engineering（电子信息与电气工程学院）；Li Auto（力汽车）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生实验室）；Ningbo Institute of Digital Twin（宁波数字孪生研究院）
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；IIIS, Tsinghua University（清华大学人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波工程技术院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；S-Lab, Nanyang Technological University（南洋理工大学S实验室）；ByteDance（字节跳动）；Kling, Kuaishou Technology（Kling，快手科技）；GigaAI（极佳科技）；AIR, Tsinghua University（清华大学人工智能研究院）
