Agent-SafetyBench：评估大语言模型智能体的安全性
Agent-SafetyBench: Evaluating the Safety of LLM Agents
AI总结：
针对大语言模型智能体安全评估缺乏综合基准的问题，本文推出包含349个交互环境、2000个测试用例的Agent-SafetyBench基准，测试发现16款主流智能体安全得分均未超60%，指出其鲁棒性与风险意识不足的缺陷，并开源基准以推动相关研究。
摘要：
随着大语言模型（LLM）越来越多地被部署为智能体，它们与交互环境的集成以及工具的使用带来了超出模型本身相关问题的全新安全挑战。然而，目前缺乏用于评估智能体安全性的综合基准，这对有效评估和进一步改进构成了重大障碍。本文中，我们推出了Agent-SafetyBench，这是一款专为评估LLM智能体安全性设计的综合基准。Agent-SafetyBench包含349个交互环境和2000个测试用例，可评估8类安全风险，覆盖不安全交互中常见的10种典型失败模式。我们对16款主流LLM智能体的评估得出了令人担忧的结果：没有任何一款智能体的安全得分超过60%。这凸显了LLM智能体面临的严峻安全挑战，也表明该领域存在巨大的改进需求。通过失败模式与有用性分析，我们总结出当前LLM智能体存在的两大根本安全缺陷：鲁棒性不足和风险意识缺失。此外，我们的研究结果表明，仅依赖防御提示可能不足以解决这些安全问题，这凸显了研发更先进、更鲁棒策略的必要性。为推动该领域的发展，Agent-SafetyBench已在https://github.com/thu-coai/Agent-SafetyBench/发布，以促进智能体安全评估与改进方向的后续研究。
英文摘要：
As large language models (LLMs) are increasingly deployed as agents, their integration into interactive environments and tool use introduce new safety challenges beyond those associated with the models themselves. However, the absence of comprehensive benchmarks for evaluating agent safety presents a significant barrier to effective assessment and further improvement. In this paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%. This highlights significant safety challenges in LLM agents and underscores the considerable need for improvement. Through failure mode and helpfulness analysis, we summarize two fundamental safety defects in current LLM agents: lack of robustness and lack of risk awareness. Furthermore, our findings suggest that reliance on defense prompts alone may be insufficient to address these safety issues, emphasizing the need for more advanced and robust strategies. To drive progress in this area, Agent-SafetyBench has been released at https://github.com/thu-coai/Agent-SafetyBench/ to facilitate further research in agent safety evaluation and improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
