Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.
Benchmarking LLM Judges for Mobile Agent Evaluation
Mila – Québec AI Institute(米拉-魁北克人工智能研究所);Concordia University(康考迪亚大学);University of Toronto(多伦多大学);Shanghai University(上海大学);McMaster University(麦克马斯特大学)
HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Zhejiang University(浙江大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Pretraining Team, xAI(预训练团队,xAI);Hong Kong University of Science and Technology (GuangZhou)(香港科技大学(广州))
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Technical University of Munich (TUM)(慕尼黑技术大学);Ludwig Maximilian University of Munich (LMU)(慕尼黑路德维希-马克西米利安大学);University of Konstanz(康斯坦茨大学);University of Saarbrücken(萨尔布吕肯大学)
Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation
School of Computer Science and Engineering, The Hebrew University of Jerusalem(计算机科学与工程学院,特拉维夫大学);IBM Research(IBM研究院);The Allen Institute for AI (AI2)(人工智能研究所)
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University(清华大学对话人工智能(CoAI)小组);Zhipu AI(智谱AI);University of Electronic Science and Technology of China(电子科技大学)
Med-RewardBench: Benchmarking Reward Models and Judges for Medical Multimodal Large Language Models
College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机科学与软件工程学院);School of Artificial Intelligence, Shenzhen University(深圳大学人工智能学院);Guangdong Provincial Key Laboratory of Intelligent Information Processing(广东省智能信息处理重点实验室);The Hong Kong University of Science and Technology(香港科学与技术大学);Renmin University of China(中国人民大学);National Yang Ming Chiao Tung University;Taipei Veterans General Hospital(台北荣民总医院);School of Biomedical Engineering, Shenzhen University(深圳大学生物医学工程学院)
Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs
Renmin University of China(中国人民大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Chinese University of Hong Kong(香港中文大学);Johns Hopkins University(约翰霍普金斯大学);Hong Kong University of Science and Technology(香港科技大学);Tencent(腾讯)
Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness
Division of Computational Health Sciences, Department of Surgery University of Minnesota(大学医学计算科学系,外科系);Institute for Health Informatics University of Minnesota(健康信息学研究所,明尼苏达大学);Department of Electrical and Computer Engineering University of Minnesota(电气与计算机工程系,明尼苏达大学)
CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs
MATS;Mila – Quebec AI Institute(Mila – 魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Astra Fellowship;ELLIS Institute Tübingen, MPI for Intelligent Systems & Tübingen AI Center;LawZero;Google DeepMind(谷歌DeepMind)