Existing top-performance autonomous driving systems typically rely on the multi-modal fusion strategy for reliable scene understanding. This design is however fundamentally restricted due to overlooking the modality-specific strengths and finally hampering the model performance. To address this limitation, in this work, we introduce a novel modality interaction strategy that allows individual per-modality representations to be learned and maintained throughout, enabling their unique characteristics to be exploited during the whole perception pipeline. To demonstrate the effectiveness of the proposed strategy, we design DeepInteraction++, a multi-modal interaction framework characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Specifically, the encoder is implemented as a dual-stream Transformer with specialized attention operation for information exchange and integration between separate modality-specific representations. Our multi-modal representational learning incorporates both object-centric, precise sampling-based feature alignment and global dense information spreading, essential for the more challenging planning task. The decoder is designed to iteratively refine the predictions by alternately aggregating information from separate representations in a unified modality-agnostic manner, realizing multi-modal predictive interaction. Extensive experiments demonstrate the superior performance of the proposed framework on both 3D object detection and end-to-end autonomous driving tasks. Our code is available at https://github.com/fudan-zvg/DeepInteraction.
MATS: A novel multi-modality multi-task learning framework for 3D perception in autonomous driving
School of Electrical and Information Engineering, Zhengzhou University（郑州大学电气与信息工程学院）；Tianping College of Suzhou University of Science and Technology（苏州科技大学天平学院）；Toronto Metropolitan University（多伦多都会大学）
Generating Multimodal Driving Scenes via Next-Scene Prediction
School of Software Engineering, XJTU（西安交通大学软件学院）；Horizon Robotics（地平线机器人）；School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）；University of Chinese Academy of Sciences（中国科学院大学）
Self-Supervised Bird's Eye View Motion Prediction with Cross-Modality Signals
Shanghai Jiao Tong University（上海交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；University of Southern California（南加州大学）；Shanghai AI Laboratory（上海人工智能实验室）
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑科学智能科技研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）
Mutual Modality Trust with Lightweight Reconstruction Regularization for Fine-grained Tire Pattern Recognition
School of Communications and Information Engineering, Xi’an University of Posts and Telecommunications（西安邮电大学通信与信息工程学院）；School of Artificial Intelligence, Hainan Normal University（海南师范大学人工智能学院）；School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能与光电学院）
Wayformer: Motion Forecasting via Simple & Efficient Attention Networks
Waymo（Waymo公司）
自动驾驶的运动预测是一项具有挑战性的任务，因为复杂的驾驶场景会导致静态和动态输入的异构混合。这是一个开放的问题，如何最好地表示和融合信息的道路几何形状，车道的连通性，时变的交通灯状态，和历史的一组动态代理和他们的相互作用到一个有效的编码。为了对这组不同的输入特征进行建模，提出了许多方法来设计具有不同的模态特定模块的同样复杂的系统。这导致系统很难以严格的方式进行扩展、扩展或调整，以权衡质量和效率。在本文中，我们提出了Wayformer，一个家庭的注意力为基础的架构，运动预测是简单和均匀的。Wayformer提供了一个紧凑的模型描述，包括一个基于注意力的场景编码器和一个解码器。在场景编码器中，我们研究了输入模态的早期、晚期和分层融合的选择。对于每一种融合类型，我们探索的策略，权衡效率和质量，通过因子化的注意力或潜在的查询注意力。我们表明，尽管早期融合的构造简单，但它不仅是模态不可知的，而且在Waymo Open MotionDataset（WOMD）和Argoverse排行榜上都取得了最先进的结果，证明了我们设计理念的有效性
Institute of Trustworthy Embodied AI, Fudan University（可信具身人工智能研究院，复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Key Laboratory of Target Cognition and Application Technology, Aerospace Information Research Institute, Chinese Academy of Sciences（目标认知与应用技术重点实验室，航天信息研究所，中国科学院）；University of Science and Technology of China（中国科学技术大学）
OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Westlake University（西湖大学）；Li Auto Inc.（Li汽车公司）；The University of Toronto（多伦多大学）；University of Macau（澳门大学）
MM-Future: Multi-Mode Joint World-Action Modeling for Autonomous Driving
NIO（蔚来）；Artificial General Intelligence Institute, University of Science and Technology of China（中国科学技术大学人工智能研究院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Beihang University（北京航空航天大学）
DriveFine: Refining-Augmented Masked Diffusion VLA for Precise and Robust Driving
Huazhong University of Science and Technology（华中科技大学）；Xiaomi EV（小米电动车）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）
DepthVision: Enabling Robust Vision-Language Models with GAN-Based LiDAR-to-RGB Synthesis for Autonomous Driving
Chair of Robotics, Artificial Intelligence and Real-time Systems, Technical University of Munich（机器人学、人工智能与实时系统教授会，慕尼黑技术大学）；Computer Science and Engineering Department, University of California Merced（计算机科学与工程系，加州大学默塞德分校）
