The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system design. Traditional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM's reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by leveraging a traditional end-to-end driving model to produce condensed and semantically enriched representations of the scene, which are optimized for LLM planning compatibility through deliberate representation and reasoning alignment training stages. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning.
Learning Rollout from Sampling:An R1-Style Tokenized Traffic Simulation Model
State Key Laboratory of Intelligent Transportation System, Key Laboratory of Autonomous Transportation Technology for Special Vehicles, Ministry of Industry and Information Technology, School of Transportation Science and Engineering, Beihang University(北京航空航天大学交通科学与工程学院,智能交通系统国家重点实验室,特种车辆自主运输技术重点实验室(工业和信息化部));State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所,多模态人工智能系统国家重点实验室)
LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarios
Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Horizon Robotics(地平线机器人)
Department of Civil and Environmental Engineering, University of Wisconsin–Madison(威斯康星大学麦迪逊分校土木与环境工程系);College of Computing and Data Science, Nanyang Technological University(南洋理工大学 computing and Data Science 学院);College of Transportation, Tongji University(同济大学交通运输学院)
Zero-Human Demonstration End-to-end Autonomous Driving with Trajectory Scorer
Institute of Trustworthy Embodied AI(可信具身人工智能研究所);Fudan University(复旦大学);Shanghai Key Laboratory of Multimodal Embodied AI(上海多模态具身人工智能重点实验室);NVIDIA;University of Michigan(密歇根大学);East China Normal University(华东师范大学)
RiskNet: Interaction-Aware Risk Forecasting for Autonomous Driving in Long-Tail Scenarios
School of Transportation, Southeast University(东南大学交通学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院)
RADE: Learning Risk-Adjustable Driving Environment via Multi-Agent Conditional Diffusion
Department of Civil and Environmental Engineering, University of Michigan(密歇根大学土木与环境工程系);Department of Computer Science, University of Hong Kong(香港大学计算机科学系);University of Michigan Transportation Research Institute(密歇根大学交通研究研究院)
Prospective Role of Foundation Models in Advancing Autonomous Vehicles
School of Automotive Studies, Tongji University(同济大学汽车学院);Frontiers Science Center for Intelligent Autonomous Systems, Tongji University(同济大学智能自主系统前沿科学中心);SAIC Intelligent Technology(上汽智能科技公司);College of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Research & Advanced Engineering, Ford Motor Company(福特汽车公司研发与高级工程部门);College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院);National Key Laboratory of Autonomous Intelligent Unmanned Systems(自主智能无人系统全国重点实验室)