Evaluation of Safety Cognition Capability in Vision-Language Models for Autonomous Driving
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）
Ensuring the safety of vision-language models (VLMs) in autonomous driving systems is of paramount importance, yet existing research has largely focused on conventional benchmarks rather than safety-critical evaluation. In this work, we present SCD-Bench (Safety Cognition Driving Benchmark) a novel framework specifically designed to assess the safety cognition capabilities of VLMs within interactive driving scenarios. To address the scalability challenge of data annotation, we introduce ADA (Autonomous Driving Annotation), a semi-automated labeling system, further refined through expert review by professionals with domain-specific knowledge in autonomous driving. To facilitate scalable and consistent evaluation, we also propose an automated assessment pipeline leveraging large language models, which demonstrates over 98% agreement with human expert judgments. In addressing the broader challenge of aligning VLMs with safety cognition in driving environments, we construct SCD-Training, the first large-scale dataset tailored for this task, comprising 324.35K high-quality samples. Through extensive experiments, we show that models trained on SCD-Training exhibit marked improvements not only on SCD-Bench, but also on general and domain-specific benchmarks, offering a new perspective on enhancing safety-aware interactions in vision-language systems for autonomous driving.
What is the Visual Cognition Gap between Humans and Multimodal LLMs?
Department of Computer Science, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；College of Computing, Georgia Institute of Technology（佐治亚理工学院计算机学院）；Department of Computer Science, University of Virginia（弗吉尼亚大学计算机科学系）；Digital Twin Lab, Purdue University（普渡大学数字孪生实验室）；HKUST (Guangzhou)（香港科技大学（广州））；Department of Rehabilitation Medicine, Shenzhen Children’s Hospital（深圳儿童医院康复医学系）
From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
School of Information Science, Beijing Language and Culture University（北京语言大学信息科学学院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreaks
Department of Computer Science（计算机科学系）；Luddy School of Informatics, Computing, and Engineering（信息学、计算与工程学院）；Indiana University Bloomington（印第安纳大学布卢明顿分校）
From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京校区人工智能学院）；Xiamen University（厦门大学）；The Hong Kong University of Science and Technology（香港理工大学）；Nanyang Technological University（南洋理工大学）
DM-KG: A Novel Method for Boosting Spatial Cognition of Vision-Language Models in Street View Imagery
Institute of Surveying and Mapping, Information Engineering University（信息工程大学测绘学院）；Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences（中国科学院地理科学与资源研究所）
SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
Beihang University（北航）；Beijing Institute of Technology（北京理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
HoloAegis: Frozen Representation, Topological Inference --- Minimally Parametric Safety Manifolds and Their Capability Boundaries for LLM Guardrails
Hong Kong Baptist University（香港浸会大学）；Guangdong Polytechnic Normal University（广东技术师范大学）；Guangdong Institute of Digital Industry（广东数字产业研究院）；The Hong Kong Polytechnic University（香港理工大学）；The Education University of Hong Kong（香港教育大学）；Southern University of Science and Technology（南方科技大学）
Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
Tongji University（同济大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；College of Cryptology and Cyber Science, Nankai University（南开大学密码与网络科学学院）
RailVQA: A Benchmark and Framework for Efficient Interpretable Visual Cognition in Automatic Train Operation
School of Automation and Intelligence, Beijing Jiaotong University（北京交通大学自动化与智能学院）；School of Software, Northwestern Polytechnical University（西北工业大学软件学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
What lies between `\emph{sensing}' and `\emph{sensibility}'? In other words, what kind of cognitive processes mediate sensing capability, and the formation of sensible impressions ---e.g., abstractions, analogies, hypotheses and theory formation, beliefs and their revision, argument formation--- in domain-specific problem solving, or in regular activities of everyday living, working and simply going around in the environment? How can knowledge and reasoning about such capabilities, as exhibited by humans in particular problem contexts, be used as a model and benchmark for the development of co…
A Framework for a Capability-driven Evaluation of Scenario Understanding for Multimodal Large Language Models in Autonomous Driving
Dr. Ing. h.c. F. Porsche AG（保时捷股份公司）；Forschungszentrum Informatik（弗劳恩霍夫信息技术研究中心）；Hochschule Esslingen（埃斯林根应用技术大学）；University of Michigan（密歇根大学）；Voxel51；Karlsruher Institut für Technologie（卡尔斯鲁厄理工学院）
Exploring Adversarial Robustness and Safety Alignment in Multilingual Multi-Modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（卡比拉大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
