Evaluating Proactive Risk Awareness of Large Language Models
The Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));The Hong Kong Polytechnic University, Hong Kong(香港理工大学);Southern University of Science and Technology, Shenzhen(南方科技大学);Shenzhen Loop Area Institute, Shenzhen, China(深圳南山区研究院)
As large language models (LLMs) are increasingly embedded in everyday decision-making, their safety responsibilities extend beyond reacting to explicit harmful intent toward anticipating unintended but consequential risks. In this work, we introduce a proactive risk awareness evaluation framework that measures whether LLMs can anticipate potential harms and provide warnings before damage occurs. We construct the Butterfly dataset to instantiate this framework in the environmental and ecological domain. It contains 1,094 queries that simulate ordinary solution-seeking activities whose responses may induce latent ecological impact. Through experiments across five widely used LLMs, we analyze the effects of response length, languages, and modality. Experimental results reveal consistent, significant declines in proactive awareness under length-restricted responses, cross-lingual similarities, and persistent blind spots in (multimodal) species protection. These findings highlight a critical gap between current safety alignment and the requirements of real-world ecological responsibility, underscoring the need for proactive safeguards in LLM deployment.
Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration
National University of Singapore(新加坡国立大学);Singapore Management University(新加坡管理大学);The Chinese University of Hong Kong(香港中文大学);Sichuan University(四川大学)
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China(计算与智能研究所,哈尔滨工业大学,深圳,中国);Peng Cheng Laboratory, Shenzhen, China(鹏城实验室,深圳,中国)
Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
Barcelona Supercomputing Center, Spain(巴塞罗那超级计算中心);Universitat Politècnica de Catalunya, Spain(加泰罗尼亚理工大学);DFKI GmbH, Saarland Informatics Campus, Saarbrücken, Germany(DFKI GmbH萨尔兰信息技术校区)
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国)
Large Language Models Lack Temporal Awareness of Medical Knowledge
University of Virginia(弗吉尼亚大学);National Institutes of Health(美国国家卫生研究院);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Yale University(耶鲁大学);Weill Cornell Medicine(韦氏 Cornell 医学院)
Towards Evaluating Proactive Risk Awareness of Multimodal Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Xiaohongshu Inc.(小红书公司);Renmin University of China(中国人民大学);Johns Hopkins University(约翰霍普金斯大学)
University of Mannheim(曼海姆大学);University of Technology Nuremberg(纽伦堡技术大学);National Institute of Information and Communications Technology(信息与通信技术国家研究所)
CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement
Shanghai Institute of AI for Education(上海人工智能教育研究院);School of Computer Science(计算机科学学院);East China Normal University(东华大学);Tencent Inc.(腾讯公司);Shanghai Innovation Institute(上海创新研究院)
Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(教育部知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language
Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR IHPC)(高性能计算研究所,科技研究局(A*STAR IHPC));National University of Singapore(国立新加坡大学);Shanghai University(上海大学)
Equipping Retrieval-Augmented Large Language Models with Document Structure Awareness
School of Computer Science, Beijing Institute of Technology(计算机学院,北京理工大学);Ant Group(蚂蚁集团);Southeast Academy of Information Technology, Beijing Institute of Technology(信息科技东南学院,北京理工大学)
Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
Renmin University of China(中国人民大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰霍普金斯大学);Hong Kong University of Science and Technology(香港科技大学);Tencent(腾讯)
AwarenessBench: Assessing Cognitive Capabilities of Language Models
Tsinghua University(清华大学);Shanghai Qi Zhi Institute(上海期智研究院);Carnegie Mellon University(卡内基梅隆大学);Columbia University(哥伦比亚大学);Fangcun AI(方寸智能);University of Chinese Academy of Sciences(中国科学院大学);ShanghaiTech University(上海科技大学)
The Safety Reminder: A Soft Prompt to Reactivate Delayed Safety Awareness in Vision-Language Models
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院);School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算与信息系统学院)