Semantics-Guided Multimodal Masked Autoencoder Pretraining for 3D BEV Object Detection
University of Lincoln, Lincoln Centre for Autonomous Systems（林肯大学，林肯自主系统中心）；University of Cambridge, Institute for Manufacturing, Department of Engineering（剑桥大学，制造研究所，工程系）
Prabuddhi Wariyapperuma, Rajitha de Silva, Marc Hanheide, Thomas Bohné, Leonardo Guevara
备注：
Accepted at the ICRA 2026 Workshop on Semantics for Reliable Robot Autonomy (SRRA) as a lightning talk and poster
Accurate 3D bird's-eye view (BEV) object detection is essential for autonomous driving, and depends strongly on effective multimodal representations from complementary sensors such as cameras and LiDAR. Multimodal masked autoencoders have shown strong potential for learning such representations for downstream 3D BEV object detection. However, existing methods typically apply uniform random masking to camera and LiDAR inputs, treating all regions equally, and learn representations only through masked reconstruction. We propose a semantics-guided multimodal masked autoencoder framework that introduces semantic information during pretraining through two separate components: (i) semantics-guided LiDAR voxel masking, which preserves semantically important LiDAR regions more strongly, and (ii) an auxiliary point-wise LiDAR semantic decoder branch that injects semantic guidance in addition to reconstruction. On BEVFusion 3D object detection, our semantics-guided pretraining strategy improves performance on the nuScenes mini validation set compared to the standard UniM2AE baseline: semantics-guided LiDAR voxel masking yields +1.49% mean Average Precision (mAP) and +1.66% nuScenes Detection Score (NDS), while decoder-side point semantic supervision yields +1.39% mAP and +3.22% NDS over the baseline.
M-BEV: Masked BEV Perception for Robust Autonomous Driving
Shenzhen Institute of Advanced Technology, Chinese Academy of Science（中国科学院深圳先进技术研究院）；University of Chinese Academy of Science（中国科学院大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
UniM$^2$AE: Multi-modal Masked Autoencoders with Unified 3D Representation for 3D Perception in Autonomous Driving
Harbin Institute of Technology（哈尔滨工业大学）；Tianyijiaotong Technology Ltd.（天宜交通科技有限公司）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
BEVPose: Unveiling Scene Semantics through Pose-Guided Multi-Modal BEV Alignment
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；The University of Adelaide（阿德莱德大学）；Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
LetsMap: Unsupervised Representation Learning for Semantic BEV Mapping
University of Freiburg（弗莱堡大学）；Qualcomm SARL France（法国高通公司）；QT Technologies Ireland Limited（爱尔兰QT技术有限公司）；Federal University of Rio Grande（里奥格兰德联邦大学）；University of Technology Nuremberg（纽伦堡工业大学）
MIM4D: Masked Modeling with Multi-View Video for Autonomous Driving Representation Learning
School of EIC, Huazhong University of Science & Technology（华中科技大学电子信息与通信学院）；Institute of Artificial Intelligence, Huazhong University of Science & Technology（华中科技大学人工智能研究院）；Horizon Robotics
This paper studies the BERT pretraining of video transformers. It is a straightforward but worth-studying extension given the recent success from BERT pretraining of image transformers. We introduce BEVT which decouples video representation learning into spatial representation learning and temporal dynamics learning. In particular, BEVT first performs masked image modeling on image data, and then conducts masked image modeling jointly with masked video modeling on video data. This design is motivated by two observations: 1) transformers learned on image datasets provide decent spatial priors t…
OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks
ISIR, Sorbonne Université（索邦大学ISIR研究所）；Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague（布拉格捷克理工大学捷克信息学、机器人学与控制论研究所）；FEE CTU（捷克技术大学电气工程系）；CIIRC CTU（捷克技术大学捷克信息学、机器人学与控制论研究所）；Kyutai（Kyutai公司）；LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS（法国国家科学研究中心-古斯塔夫·埃菲尔大学巴黎路桥学院LIGM实验室）
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）
The Chinese University of Hong Kong（香港中文大学）；NVIDIA AI Technology Center（英伟达人工智能技术中心）；SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）
FlowFormer introduces a transformer architecture into optical flow estimation and achieves state-of-the-art performance. The core component of FlowFormer is the transformer-based cost-volume encoder. Inspired by the recent success of masked autoencoding (MAE) pretraining in unleashing transformers' capacity of encoding visual representation, we propose Masked Cost Volume Autoencoding (MCVA) to enhance FlowFormer by pretraining the cost-volume encoder with a novel MAE scheme. Firstly, we introduce a block-sharing masking strategy to prevent masked information leakage, as the cost maps of neighb…
Diffusion Masked Pretraining for Dynamic Point Cloud
Xi’an Jiaotong University（西安交通大学）；School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）；University of Western Australia（西澳大学）
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；Monash University（莫纳什大学）；Nile University（尼罗河大学）；Nanyang Technological University（南洋理工大学）
Understanding Self-Supervised Pretraining with Part-Aware Representation Learning
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China（高可信软件技术教育部重点实验室（北京大学））；School of Computer Science, Peking University（北京大学计算机学院）；School of AI, Peking University（北京大学人工智能学院）；University of Hong Kong（香港大学）；School of EIC, Huazhong University of Science & Technology（华中科技大学电子信息与通信学院）；Baidu（百度）；UC Berkeley（加州大学伯克利分校）
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental r…
Masked autoencoding has achieved great success for self-supervised learning in the image and language domains. However, mask based pretraining has yet to show benefits for point cloud understanding, likely due to standard backbones like PointNet being unable to properly handle the training versus testing distribution mismatch introduced by masking during training. In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. Our key idea is to represent the point cloud as discrete occupancy values (1 if part of the point c…
University of Science and Technology of China（中国科学技术大学）；Microsoft Research Asia（微软亚洲研究院）
We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining. The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance. Therefore, w…
Deblurring Masked Autoencoder is Better Recipe for Ultrasound Image Recognition
West China Hospital, Sichuan University（四川大学华西医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Sichuan University（四川大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Self-attention based transformer models have been dominating many computer vision tasks in the past few years. Their superb model qualities heavily depend on the excessively large labeled image datasets. In order to reduce the reliance on large labeled datasets, reconstruction based masked autoencoders are gaining popularity, which learn high quality transferable representations from unlabeled images. For the same purpose, recent weakly supervised image pretraining methods explore language supervision from text captions accompanying the images. In this work, we propose masked image pretraining…
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）
Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computat…
Emory University（埃默里大学）；University of Southern California（南加州大学）；Arizona State University（亚利桑那州立大学）；Mayo Clinic（妙佑医疗国际）
Language modality within the vision language pretraining framework is innately discretized, endowing each word in the language vocabulary a semantic meaning. In contrast, visual modality is inherently continuous and high-dimensional, which potentially prohibits the alignment as well as fusion between vision and language modalities. We therefore propose to "discretize" the visual representation by joint learning a codebook that imbues each visual token a semantic. We then utilize these discretized visual semantics as self-supervised ground-truths for building our Masked Image Modeling objective…
$\mathsf{CSMAE~}$:~Cataract Surgical Masked Autoencoder (MAE) based Pre-training
Johns Hopkins University（约翰斯·霍普金斯大学）；Wilmer Eye Institute, Johns Hopkins University（约翰斯·霍普金斯大学威尔默眼科研究所）；Malone Center for Engineering in Healthcare, Johns Hopkins University（约翰斯·霍普金斯大学马龙医疗保健工程中心）
