Towards Evaluating Proactive Risk Awareness of Multimodal Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Xiaohongshu Inc.(小红书公司);Renmin University of China(中国人民大学);Johns Hopkins University(约翰霍普金斯大学)
Human safety awareness gaps often prevent the timely recognition of everyday risks. In solving this problem, a proactive safety artificial intelligence (AI) system would work better than a reactive one. Instead of just reacting to users' questions, it would actively watch people's behavior and their environment to detect potential dangers in advance. Our Proactive Safety Bench (PaSBench) evaluates this capability through 416 multimodal scenarios (128 image sequences, 288 text logs) spanning 5 safety-critical domains. Evaluation of 36 advanced models reveals fundamental limitations: Top performers like Gemini-2.5-pro achieve 71% image and 64% text accuracy, but miss 45-55% risks in repeated trials. Through failure analysis, we identify unstable proactive reasoning rather than knowledge deficits as the primary limitation. This work establishes (1) a proactive safety benchmark, (2) systematic evidence of model limitations, and (3) critical directions for developing reliable protective AI. We believe our dataset and findings can promote the development of safer AI assistants that actively prevent harm rather than merely respond to requests. Our dataset can be found at https://huggingface.co/datasets/Youliang/PaSBench.
Evaluating Proactive Risk Awareness of Large Language Models
The Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong Polytechnic University, Hong Kong(香港理工大学);Southern University of Science and Technology, Shenzhen(南方科技大学);Shenzhen Loop Area Institute, Shenzhen, China(深圳南山区研究院)
The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算与信息系统学院)
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能研究所,哈尔滨工业大学,深圳,中国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);The Chinese University of Hong Kong(香港中文大学);Sichuan University(四川大学)
AwarenessBench: Assessing Cognitive Capabilities of Language Models
Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院);Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);Fangcun AI(方寸智能);University of Chinese Academy of Sciences(中国科学院大学);ShanghaiTech University(上海科技大学)
Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
Barcelona Supercomputing Center, Spain(巴塞罗那超级计算中心);Universitat Politècnica de Catalunya, Spain(加泰罗尼亚理工大学);DFKI GmbH, Saarland Informatics Campus, Saarbrücken, Germany(DFKI GmbH萨尔兰信息技术校区)
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国)
Large Language Models Lack Temporal Awareness of Medical Knowledge
University of Virginia(弗吉尼亚大学);National Institutes of Health(美国国家卫生研究院);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Yale University(耶鲁大学);Weill Cornell Medicine(韦氏 Cornell 医学院)
ProMediConv: Benchmarking Proactive Conversational Agents in Legal Dispute Mediation
University of Science and Technology of China(中国科学技术大学);Singapore Management University(新加坡管理大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
Xi’an Jiaotong University(西安交通大学);ARC Lab, Tencent IEG(腾讯IEG ARC实验室);City University of Hong Kong(香港城市大学);Institute of Automation, CAS(中国科学院自动化研究所);Harvard University(哈佛大学);Nanjing University(南京大学)
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language
Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR IHPC)(高性能计算研究所,科技研究局(A*STAR IHPC));National University of Singapore(国立新加坡大学);Shanghai University(上海大学)
Equipping Retrieval-Augmented Large Language Models with Document Structure Awareness
School of Computer Science, Beijing Institute of Technology(计算机学院,北京理工大学);Ant Group(蚂蚁集团);Southeast Academy of Information Technology, Beijing Institute of Technology(信息科技东南学院,北京理工大学)
Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
Renmin University of China(中国人民大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰霍普金斯大学);Hong Kong University of Science and Technology(香港科技大学);Tencent(腾讯)