Recent advancements in vision foundation models (VFMs) have revolutionized visual perception in 2D, yet their potential for 3D scene understanding, particularly in autonomous driving applications, remains underexplored. In this paper, we introduce LargeAD, a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. This alignment facilitates cross-modal representation learning, enhancing the semantic consistency between 2D and 3D data. We introduce several key innovations: (i) VFM-driven superpixel generation for detailed semantic representation, (ii) a VFM-assisted contrastive learning strategy to align multimodal features, (iii) superpoint temporal consistency to maintain stable representations across time, and (iv) multi-source data pretraining to generalize across various LiDAR configurations. Our approach achieves substantial gains over state-of-the-art methods in linear probing and fine-tuning for LiDAR-based segmentation and object detection. Extensive experiments on 11 large-scale multi-sensor datasets highlight our superior performance, demonstrating adaptability, efficiency, and robustness in real-world autonomous driving scenarios.
The Chinese University of Hong Kong（香港中文大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Deep visuomotor policy learning, which aims to map raw visual observation to action, achieves promising results in control tasks such as robotic manipulation and autonomous driving. However, it requires a huge number of online interactions with the training environment, which limits its real-world application. Compared to the popular unsupervised feature learning for visual recognition, feature pretraining for visuomotor control tasks is much less explored. In this work, we aim to pretrain policy representations for driving tasks by watching hours-long uncurated YouTube videos. Specifically, w…
Korea Advanced Institute of Science and Technology（韩国科学技术院）
We present a training pipeline for the autonomous driving task given the current camera image and vehicle speed as the input to produce the throttle, brake, and steering control output. The simulator Airsim's convenient weather and lighting API provides a sufficient diversity during training which can be very helpful to increase the trained policy's robustness. In order to not limit the possible policy's performance, we use a continuous and deterministic control policy setting. We utilize ResNet-34 as our actor and critic networks with some slight changes in the fully connected layers. Conside…
Enhanced Spatiotemporal Consistency for Image-to-LiDAR Data Pretraining
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）；School of Computing, Department of Computer Science, National University of Singapore（新加坡国立大学计算机学院）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Shanghai AI Laboratory（上海人工智能实验室）；S-Lab, Nanyang Technological University（南洋理工大学S实验室）
Norwegian University of Science and Technology（挪威科技大学）
The need for simulated data in autonomous driving applications has become increasingly important, both for validation of pretrained models and for training new models. In order for these models to generalize to real-world applications, it is critical that the underlying dataset contains a variety of driving scenarios and that simulated sensor readings closely mimics real-world sensors. We present the Carla Automated Dataset Extraction Tool (CADET), a novel tool for generating training data from the CARLA simulator to be used in autonomous driving research. The tool is able to export high-quali…
Saliency-Guided Domain Adaptation for Left-Hand Driving in Autonomous Steering
Centre for Accident Research and Road Safety-Queensland, Queensland University of Technology（昆士兰事故研究与道路安全中心、昆士兰理工大学）；QUT Centre for Robotics and School of Electrical Engineering and Computer Science, Queensland University of Technology（昆士兰大学机器人中心及电气工程与计算机科学学院、昆士兰理工大学）
CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion
Wuhan University of Technology（武汉理工大学）；University of North Carolina at Charlotte（北卡罗来纳州立大学）；Wuhan University（武汉大学）；Southwest Jiaotong University（西南交通大学）；The Hong Kong Polytechnic University（香港理工大学）
The University of Sydney（悉尼大学）；SenseTime Research（商汤科技研究院）；Zhejiang University（浙江大学）；Shanghai AI Laboratory（上海人工智能实验室）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）
Human-centric perceptions include a variety of vision tasks, which have widespread industrial applications, including surveillance, autonomous driving, and the metaverse. It is desirable to have a general pretrain model for versatile human-centric downstream tasks. This paper forges ahead along this path from the aspects of both benchmark and pretraining methods. Specifically, we propose a \textbf{HumanBench} based on existing datasets to comprehensively evaluate on the common ground the generalization abilities of different pretraining methods on 19 datasets from 6 diverse downstream tasks, i…
Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. Th…
Beyond One Shot, Beyond One Perspective: Cross-View and Long-Horizon Distillation for Better LiDAR Representations
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）
CHARMS: A Cognitive Hierarchical Agent for Reasoning and Motion Stylization in Autonomous Driving
School of Mechanical and Electronic Engineering, Wuhan University of Technology（武汉理工大学机械与电子工程学院）；Intelligent Transportation Systems Research Center, Wuhan University of Technology（武汉理工大学智能交通系统研究所）；Department of Data and Systems Engineering, The University of Hong Kong（香港大学数据与系统工程系）；School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；School of Mechanical Engineering, Southeast University（东南大学机械工程学院）
Road scene understanding tasks have recently become crucial for self-driving vehicles. In particular, real-time semantic segmentation is indispensable for intelligent self-driving agents to recognize roadside objects in the driving area. As prior research works have primarily sought to improve the segmentation performance with computationally heavy operations, they require far significant hardware resources for both training and deployment, and thus are not suitable for real-time applications. As such, we propose a doubly contrastive approach to improve the performance of a more practical ligh…
Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems（林肯大学，林肯自主系统中心）；University of Cambridge, Institute for Manufacturing, Department of Engineering（剑桥大学，制造研究所，工程系）
Exploring the Untouched Sweeps for Conflict-Aware 3D Segmentation Pretraining
School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Department of Computer Science and Technology, Xiamen University（厦门大学计算机科学与技术系）
Toward Unified Multimodal Representation Learning for Autonomous Driving
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA（德克萨斯大学机械工程系，德克萨斯农工大学，学院站，德克萨斯，77843，美国）；The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA（工程技术与工业分布系，德克萨斯农工大学，学院站，德克萨斯，77843，美国）
GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives
XPeng Inc.（小鹏汽车）；Peking University（北京大学）；The University of Hong Kong（香港大学）；University of California, Berkeley（加州大学伯克利分校）；Princeton University（普林斯顿大学）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）；HKUST (GZ)（香港科技大学（广州））
Self-supervised pretraining is the method of choice for natural language processing models and is rapidly gaining popularity in many vision tasks. Recently, self-supervised pretraining has shown to outperform supervised pretraining for many downstream vision applications, marking a milestone in the area. This superiority is attributed to the negative impact of incomplete labelling of the training images, which convey multiple concepts, but are annotated using a single dominant class label. Although Self-Supervised Learning (SSL), in principle, is free of this limitation, the choice of pretext …
Shanghai AI Laboratory（上海人工智能实验室）；University of California at San Diego（加州大学圣迭戈分校）；Shanghai Jiao Tong University（上海交通大学）
Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving. Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the a…
E3AD: An Emotion-Aware Vision-Language-Action Model for Human-Centric End-to-End Autonomous Driving
McGill University（麦吉尔大学）；University of Macau（澳门大学）；The Hong Kong Polytechnic University（香港理工大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）
VECTOR-Drive: Tightly Coupled Vision-Language and Trajectory Expert Routing for End-to-End Autonomous Driving
College of Automotive Engineering, Jilin University（吉林大学汽车工程学院）；The National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University（吉林大学汽车底盘集成与生物力学国家级重点实验室）；ReeFocus AI Technology（ReeFocus人工智能技术）
4D Contrastive Superflows are Dense 3D Representation Learners
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nanjing University of Posts and Telecommunications（南京邮电大学）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab（注：S-Lab为南洋理工大学下属实验室，按要求提取到大学/研究院级别，此处保留原文机构名及对应翻译））
Reinforced Refinement with Self-Aware Expansion for End-to-End Autonomous Driving
School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与航空航天工程学院）；OpenDriveLab, the School of Computing and Data Science, The University of Hong Kong（开放驾驶实验室，计算与数据科学学院，香港大学）；Shanghai Innovation Institute（上海创新研究院）
SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Mercedes-Benz AG（梅赛德斯-奔驰集团）；University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；TU Munich（慕尼黑工业大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；University of Stuttgart（斯图加特大学）；UCLA（加州大学洛杉矶分校）
Evaluation Is All You Need for Multi-Modal Autonomous Driving
School of Vehicle and Mobility & College of AI, Tsinghua University（清华大学车辆与运载学院与人工智能学院）；Dongfeng Motor Corporation Research & Development Institute（东风汽车集团有限公司研发总院）
Reflect-VLA: Internalizing Reflective Guidance via Dual-Path On-Policy Self-Distillation for Autonomous Driving
Tsinghua University（清华大学）；University of Macau（澳门大学）；University of Science and Technology of China（中国科学技术大学）；Beijing Jiaotong University（北京交通大学）；Nanyang Technological University（南洋理工大学）
Learning Vision-Language-Action World Models for Autonomous Driving
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Central Research Institute, Huawei（华为中央研究院）
