本研究通过利用大型语言模型(LLM)的情境推理能力,解决自动驾驶(AD)中增强态势感知的关键需求。不同于依赖僵化、基于标签标注的传统感知系统,该研究将实时多模态传感器数据整合到统一的、LLM 可读的知识库中,使 LLM 能够动态理解并响应复杂驾驶环境。为克服 LLM 固有的延迟和模态限制,研究为自动驾驶设计了主动式检索增强生成(RAG),并结合思维链提示机制,以确保快速且上下文丰富的理解。使用真实车联万物(V2X)数据集开展的实验结果表明,感知和预测性能均获得显著提升,凸显了该框架在增强下一代 AD 系统安全性、适应性和决策能力方面的潜力。
英文摘要:
This study addresses the critical need for enhanced situational awareness in autonomous driving (AD) by leveraging the contextual reasoning capabilities of large language models (LLMs). Unlike traditional perception systems that rely on rigid, label-based annotations, it integrates real-time, multimodal sensor data into a unified, LLMs-readable knowledge base, enabling LLMs to dynamically understand and respond to complex driving environments. To overcome the inherent latency and modality limitations of LLMs, a proactive Retrieval-Augmented Generation (RAG) is designed for AD, combined with a chain-of-thought prompting mechanism, ensuring rapid and context-rich understanding. Experimental results using real-world Vehicle-to-everything (V2X) datasets demonstrate significant improvements in perception and prediction performance, highlighting the potential of this framework to enhance safety, adaptability, and decision-making in next-generation AD systems.
Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirers
National Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);Artificial Intelligence Research Institute(人工智能研究院);Shenzhen Institutes of Advanced Technology(深圳先进技术研究院);University of California, Santa Cruz(加州大学圣克鲁兹分校);University of Texas, Dallas(德克萨斯大学达拉斯分校);University of Minnesota(明尼苏达大学)
Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes
School of Information and Electronic Engineering (Sussex Artificial Intelligence Institute), Zhejiang Gongshang University(信息与电子工程学院(Sussex人工智能研究院),浙江工商大学);Sussex Artificial Intelligent Institute, Zhejiang Gongshang University(Sussex人工智能研究院,浙江工商大学);Department of Engineering and Design, University of Sussex(工程与设计系, Sussex大学);Artificial Intelligence Research Institute, Shenzhen MSU-BIT University(人工智能研究院,深圳MSU-BIT大学);College of Computer Science and Software Engineering, Shenzhen University(计算机科学与软件工程学院,深圳大学);Department of Electrical and Computer Engineering, The University of British Columbia(电气与计算机工程系,不列颠哥伦比亚大学)
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
Beijing Institute of Technology(北京理工大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院)
Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots
Department of Mechanical and Aerospace Engineering, Tandon School of Engineering, New York University(机械与航空航天工程系,坦顿工程学院,纽约大学);GenAuto.ai by General Autonomy Inc.(General Autonomy Inc. 的 GenAuto.ai)
From PowerPoint UI Sketches to Web-Based Applications: Pattern-Driven Code Generation for GIS Dashboard Development Using Knowledge-Augmented LLMs, Context-Aware Visual Prompting, and the React Framework
AccidentGPT: Accident Analysis and Prevention from V2X Environmental Perception with Multi-modal Large Model
State Key Lab of Intelligent Transportation System, Beihang University(北京航空航天大学智能交通系统国家重点实验室);Zhongguancun Laboratory(中关村实验室);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Hong Kong(香港大学)