对驾驶场景的演化进行建模对自动驾驶系统的评估与决策至关重要。现有大多数方法仅聚焦场景演化的单一维度,例如地图生成、运动预测和轨迹规划。本文提出了一个统一的Generative Pre-training for Driving(GPD-1,驾驶生成式预训练)模型,无需额外微调即可同时完成所有这些任务。我们用自车(ego)、智能体(agent)和地图(map)三类token表示每个场景,并将自动驾驶建模为一个统一的token生成问题。模型采用自回归transformer架构,使用场景级注意力掩码实现场景内的双向交互。针对自车和智能体token,我们提出分层位置tokenizer,可有效编码2D位置和航向角。针对地图token,我们训练了一个地图矢量量化自编码器,可将以自车为中心的语义地图高效压缩为离散token。我们在大规模nuPlan数据集上对GPD-1进行预训练,并开展大量实验验证其有效性。通过不同提示,GPD-1无需微调即可成功泛化到多种任务,包括场景生成、交通仿真、闭环仿真、地图预测和运动规划。代码地址:https://github.com/wzzheng/GPD。
英文摘要:
Modeling the evolutions of driving scenarios is important for the evaluation and decision-making of autonomous driving systems. Most existing methods focus on one aspect of scene evolution such as map generation, motion prediction, and trajectory planning. In this paper, we propose a unified Generative Pre-training for Driving (GPD-1) model to accomplish all these tasks altogether without additional fine-tuning. We represent each scene with ego, agent, and map tokens and formulate autonomous driving as a unified token generation problem. We adopt the autoregressive transformer architecture and use a scene-level attention mask to enable intra-scene bi-directional interactions. For the ego and agent tokens, we propose a hierarchical positional tokenizer to effectively encode both 2D positions and headings. For the map tokens, we train a map vector-quantized autoencoder to efficiently compress ego-centric semantic maps into discrete tokens. We pre-train our GPD-1 on the large-scale nuPlan dataset and conduct extensive experiments to evaluate its effectiveness. With different prompts, our GPD-1 successfully generalizes to various tasks without finetuning, including scene generation, traffic simulation, closed-loop simulation, map prediction, and motion planning. Code: https://github.com/wzzheng/GPD.
DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);East China Normal University(华东师范大学);Technical University of Munich(慕尼黑工业大学)
Data-driven Camera and Lidar Simulation Models for Autonomous Driving: A Review from Generative Models to Volume Renderers
WMG, University of Warwick(华威大学WMG学院);School of Electronics, Electrical Engineering and Computer Science at Queen’s University of Belfast(贝尔法斯特女王大学电子、电气工程与计算机科学学院)
MapDiffusion: Generative Diffusion for Vectorized Online HD Map Construction and Uncertainty Estimation in Autonomous Driving
Mercedes-Benz Research & Development North America(梅赛德斯-奔驰北美研发有限公司);University of Stuttgart(斯图加特大学);Institute for Artificial Intelligence(人工智能研究所);University of California, San Diego(加州大学圣地亚哥分校);University of Southampton(南安普顿大学)
A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems
WMG, University of Warwick(华威大学WMG学院);School of Electronics, Electrical Engineering and Computer Science at Queen’s University of Belfast(贝尔法斯特女王大学电子、电气工程与计算机科学学院)
Generative AI for Autonomous Driving: Frontiers and Opportunities
Texas A&M University(德克萨斯农工大学);University of California, Riverside(加州大学河滨分校);University of Michigan(密歇根大学);Purdue University(普渡大学);Columbia University(哥伦比亚大学);University of Florida(佛罗里达大学);Technische Universität München(慕尼黑技术大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Notre Dame(诺特丹大学);Adobe Research(Adobe研究);Lehigh University(莱斯利大学);Cleveland State University(克利夫兰州立大学);Johns Hopkins University(约翰霍普金斯大学);University of California, Merced(加州大学默塞德分校);University of Utah(犹他大学);Texas A&M Transportation Institute(德克萨斯农工大学交通研究所);Bosch Research North America(博世北美研究);Bosch Center for Artificial Intelligence (BCAI)(博世人工智能中心);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);NVIDIA(英伟达);Arizona State University(亚利桑那州立大学)
Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving
College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);COWAROBOT(酷哇机器人);School of Electronic Engineering, University of South China(南华大学电子工程学院)
TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving
LiAuto Inc(LiAuto公司);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与移动研究所)
KiGRAS: Kinematic-Driven Generative Model for Realistic Agent Simulation
University of Science and Technology of China(中国科学技术大学);Mach Drive(马赫动力);Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)
MeanFuser: Fast One-Step Multi-Modal Trajectory Generation and Adaptive Reconstruction via MeanFlow for End-to-End Autonomous Driving
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Xiaomi EV(小米汽车);Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院)
Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation
Shanghai Jiao Tong University(上海交通大学);Yinwang Intelligent Technology Co., Ltd.(银望智能科技有限公司);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Georgia Institute of Technology(佐治亚理工学院)