There is a growing interest in integrating Large Language Models (LLMs) with autonomous driving (AD) systems. However, AD systems are vulnerable to attacks against their object detection and tracking (ODT) functions. Unfortunately, our evaluation of four recent LLM agents against ODT attacks shows that the attacks are 63.26% successful in causing them to crash or violate traffic rules due to (1) misleading memory modules that provide past experiences for decision making, (2) limitations of prompts in identifying inconsistencies, and (3) reliance on ground truth perception data.
In this paper, we introduce Hudson, a driving reasoning agent that extends prior LLM-based driving systems to enable safer decision making during perception attacks while maintaining effectiveness under benign conditions. Hudson achieves this by first instrumenting the AD software to collect real-time perception results and contextual information from the driving scene. This data is then formalized into a domain-specific language (DSL). To guide the LLM in detecting and making safe control decisions during ODT attacks, Hudson translates the DSL into natural language, along with a list of custom attack detection instructions. Following query execution, Hudson analyzes the LLM's control decision to understand its causal reasoning process.
We evaluate the effectiveness of Hudson using a proprietary LLM (GPT-4) and two open-source LLMs (Llama and Gemma) in various adversarial driving scenarios. GPT-4, Llama, and Gemma achieve, on average, an attack detection accuracy of 83. 3%, 63. 6%, and 73. 6%. Consequently, they make safe control decisions in 86.4%, 73.9%, and 80% of the attacks. Our results, following the growing interest in integrating LLMs into AD systems, highlight the strengths of LLMs and their potential to detect and mitigate ODT attacks.
VTPerception-R1: Enhancing Multimodal Reasoning via Explicit Visual and Textual Perceptual Grounding
Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；Shenzhen University（深圳大学）；University of Science and Technology of China（中国科学技术大学）
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs
University of Macau（澳门大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University（东国大学计算机科学与人工智能系）；Department of Electronics and Electrical Engineering, Dongguk University（东国大学电子与电气工程系）
IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data
MARS Lab, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（MARS实验室，机械与航空航天工程学院，南洋理工大学，新加坡）；School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机械与航空航天工程学院，南洋理工大学，新加坡）
Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Tasks
Huazhong University of Science and Technology（华中科技大学）；Zhongguancun Academy（中关村学院）；East China Normal University（华东师范大学）；Zhengzhou University（郑州大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Bias
Singapore General Hospital（新加坡中央医院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Center（新加坡国家眼科中心）；Byers Eye Institute, Stanford University（斯坦福大学拜尔斯眼科研究所）；National University of Singapore（新加坡国立大学）
