LMDrive: Closed-Loop End-to-End Driving with Large Language Models
CUHK(香港中文大学);SenseTime Research(商汤科技研究院);CPII under InnoHK(InnoHK旗下CPII);University of Toronto(多伦多大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, Hongsheng Li
Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive
Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition
Institute for Intelligent Systems(智能系统研究所);Esslingen University of Applied Sciences(应用科学大学埃斯林根);Department of Computer Science(计算机科学系);University of Freiburg(弗赖堡大学)
DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);East China Normal University(华东师范大学);Technical University of Munich(慕尼黑工业大学)
ReasonPlan: Unified Scene Prediction and Decision Reasoning for Closed-loop Autonomous Driving
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所SKL-MAIS部门,北京);School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学人工智能学院,北京);EACON, Fujian, China(福建EACON机构,中国);School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, China(北京科技大学自动化与电气工程学院)
DriveE2E: Closed-Loop Benchmark for End-to-End Autonomous Driving through Real-to-Simulation
The University of Hong Kong(香港大学);AIR, Tsinghua University(清华研究院);King’s College London(伦敦国王学院);The Hong Kong Polytechnic University(香港理工大学);Tuojing Intelligence(途镜智能)
Data-Centric Evolution in Autonomous Driving: A Comprehensive Survey of Big Data System, Data Mining, and Closed-Loop Technologies
University of New South Wales(新南威尔士大学);CSIRO(澳大利亚联邦科学与工业研究组织);BYD(比亚迪);University of Notre Dame(圣母大学);The University of Sydney(悉尼大学);South China University of Technology(华南理工大学)
Hydra-NeXt: Robust Closed-Loop Driving with Open-Loop Training
Institute of Trustworthy Embodied AI, Fudan University(可信具身人工智能研究院,复旦大学);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);The Hong Kong Polytechnic University(香港理工大学);NVIDIA
PerlAD: Towards Enhanced Closed-loop End-to-end Autonomous Driving with Pseudo-simulation-based Reinforcement Learning
School of Automation and Electrical Engineering, University of Science and Technology Beijing(北京科技大学自动化与电气工程学院);Xiaomi EV(小牛电动车);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Horizon Robotics(地平线机器人)
CLOVER: Closed-Loop Value Estimation and Ranking for End-to-End Autonomous Driving Planning
Department of Automation, University of Science and Technology of China(中国科学技术大学自动化系);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院);School of Electronic Information Engineering, Beihang University(北航电子信息技术学院);National College for Excellent Engineers, Beihang University(北航卓越工程师学院)
TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving
Lomonosov Moscow State University(罗蒙诺索夫莫斯科国立大学);FusionBrain Lab(FusionBrain实验室);Moscow Institute of Physics and Technology(莫斯科物理技术学院);HSE University(高等经济大学);Innopolis University(因诺波利斯大学);NUST MISIS(国立研究型技术大学莫斯科国立钢铁合金学院)
Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations
Institute of Trustworthy Embodied AI (TEAI)(可信具身人工智能研究院);Great Wall Motor(长城汽车);Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学计算机学院及人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);University of Science and Technology of China(中国科学技术大学)
Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation
Shanghai Jiao Tong University(上海交通大学);Yinwang Intelligent Technology Co., Ltd.(银望智能科技有限公司);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Georgia Institute of Technology(佐治亚理工学院)
nuPlan-R: A Closed-Loop Planning Benchmark for Autonomous Driving via Reactive Multi-Agent Simulation
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Intelligent Transportation Thrust(智能交通方向);Robotics and Autonomous Systems Thrust(机器人与自主系统方向);Division of Emerging Interdisciplinary Areas(新兴跨学科领域部门)