Vision-language models (VLMs) have recently emerged as powerful representation learning systems that align visual observations with natural language concepts, offering new opportunities for semantic reasoning in safety-critical autonomous driving. This paper investigates how vision-language representations support driving scene safety assessment and decision-making when integrated into perception, prediction, and planning pipelines. We study three complementary system-level use cases. First, we introduce a lightweight, category-agnostic hazard screening approach leveraging CLIP-based image-text similarity to produce a low-latency semantic hazard signal. This enables robust detection of diverse and out-of-distribution road hazards without explicit object detection or visual question answering. Second, we examine the integration of scene-level vision-language embeddings into a transformer-based trajectory planning framework using the Waymo Open Dataset. Our results show that naively conditioning planners on global embeddings does not improve trajectory accuracy, highlighting the importance of representation-task alignment and motivating the development of task-informed extraction methods for safety-critical planning. Third, we investigate natural language as an explicit behavioral constraint on motion planning using the doScenes dataset. In this setting, passenger-style instructions grounded in visual scene elements suppress rare but severe planning failures and improve safety-aligned behavior in ambiguous scenarios. Taken together, these findings demonstrate that vision-language representations hold significant promise for autonomous driving safety when used to express semantic risk, intent, and behavioral constraints. Realizing this potential is fundamentally an engineering problem requiring careful system design and structured grounding rather than direct feature injection.
Using large language models for embodied planning introduces systematic safety risks
ETH Zurich（苏黎世联邦理工学院）；University College London（伦敦大学学院）；Stanford University（斯坦福大学）；Northwestern University（西北大学）；National University of Singapore（新加坡国立大学）
Safety Aware Task Planning via Large Language Models in Robotics
University of Minnesota – Twin Cities（明尼苏达大学双城分校）；Institute for Robotics and Intelligent Machines (IRIM), Georgia Institute of Technology（佐治亚理工学院机器人与智能机器研究所）；Pontificia Universidad Católica de Chile（智利天主教大学）
MedBuild AI: An Agent-Based Hybrid Intelligence Framework for Reshaping Agency in Healthcare Infrastructure Planning through Generative Design for Medical Architecture
School of Architecture and Urban Planning, Beijing University of Civil Engineering and Architecture（北京建筑大学建筑与城市规划学院）
Incorporating planning intelligence into deep learning: A planning support tool for street network design
Martin Centre for Architectural and Urban Studies, University of Cambridge（剑桥大学马丁建筑与城市研究中心）；Department of Architecture, University of Cambridge（剑桥大学建筑系）
