arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.10019cs.CLcs.AI
收藏

R-Judge: 面向大语言模型智能体的安全风险意识基准测试

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

Shanghai Jiao Tong University(上海交通大学)

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu

AI总结:

本研究提出用于评估LLM智能体行为安全风险意识的基准R-Judge,涵盖多类风险场景与交互记录,实验发现现有LLM风险意识不足,安全判断微调可有效提升性能。

摘要:

大语言模型(LLM)在各类现实应用中展现出自主完成任务的巨大潜力。尽管如此,这些LLM智能体在交互环境中运行时会带来意想不到的安全风险。过往多数研究聚焦于LLM生成内容的无害性,本工作则回应了在多样化环境中基准测试LLM智能体行为安全性的迫切需求。我们提出R-Judge,这是一个专为评估LLM基于智能体交互记录判断与识别安全风险的能力而构建的基准。R-Judge包含569条多轮智能体交互记录,覆盖5个应用类别下的27个关键风险场景以及10种风险类型。该基准经过高质量人工标注,包含安全标签与风险描述。在R-Judge上对11个LLM的评估显示,LLM的风险意识仍有很大提升空间:表现最佳的模型GPT-4o准确率仅为74.42%,其余模型均未显著超过随机水平。此外,我们发现开放智能体场景中的风险意识是一项涉及知识与推理的多维度能力,因此对LLM而言具有挑战性。进一步实验表明,针对安全判断任务的微调能显著提升模型性能,而简单的提示机制则收效甚微。R-Judge已在https://github.com/Lordog/R-Judge 公开发布。

英文摘要:

Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on the harmlessness of LLM-generated content in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks given agent interaction records. R-Judge comprises 569 records of multi-turn agent interaction, encompassing 27 key risk scenarios among 5 application categories and 10 risk types. It is of high-quality curation with annotated safety labels and risk descriptions. Evaluation of 11 LLMs on R-Judge shows considerable room for enhancing the risk awareness of LLMs: The best-performing model, GPT-4o, achieves 74.42% while no other models significantly exceed the random. Moreover, we reveal that risk awareness in open agent scenarios is a multi-dimensional capability involving knowledge and reasoning, thus challenging for LLMs. With further experiments, we find that fine-tuning on safety judgment significantly improve model performance while straightforward prompting mechanisms fail. R-Judge is publicly available at https://github.com/Lordog/R-Judge.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑