GuardAgent:通过具备知识启用的推理的Guard Agent来保护LLM智能体
GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
AI总结:
本文提出首个护栏智能体GuardAgent,通过生成任务计划并映射为护栏代码执行,动态检查目标智能体行动是否满足安全请求,结合LLM推理与记忆模块,在两个新基准上分别取得超98%和83%的护栏准确率。
摘要:
大型语言模型(LLM)智能体的快速发展引发了关于其安全性和保障的新担忧。在本文中,我们提出了GuardAgent,这是第一个通过动态检查目标智能体的行动是否满足给定的安全防护请求来保护目标智能体的护栏智能体。具体而言,GuardAgent首先分析安全防护请求以生成任务计划,然后将该计划映射为护栏代码以供执行。通过执行代码,GuardAgent能够确定性地遵循安全防护请求并保护目标智能体。在这两个步骤中,LLM被用作推理组件,并由从存储先前任务经验的记忆模块中检索到的上下文演示作为补充。此外,我们提出了两个新颖的基准:EICU-AC基准,用于评估医疗智能体的访问控制;以及Mind2Web-SC基准,用于评估Web智能体的安全策略。我们展示了GuardAgent在这两个基准上有效地调节了不同类型智能体的违规行动,护栏准确率分别超过98%和83%。项目页面:https://guardagent.github.io/
英文摘要:
The rapid advancement of large language model (LLM) agents has raised new concerns regarding their safety and security. In this paper, we propose GuardAgent, the first guardrail agent to protect target agents by dynamically checking whether their actions satisfy given safety guard requests. Specifically, GuardAgent first analyzes the safety guard requests to generate a task plan, and then maps this plan into guardrail code for execution. By performing the code execution, GuardAgent can deterministically follow the safety guard request and safeguard target agents. In both steps, an LLM is utilized as the reasoning component, supplemented by in-context demonstrations retrieved from a memory module storing experiences from previous tasks. In addition, we propose two novel benchmarks: EICU-AC benchmark to assess the access control for healthcare agents and Mind2Web-SC benchmark to evaluate the safety policies for web agents. We show that GuardAgent effectively moderates the violation actions for different types of agents on these two benchmarks with over 98% and 83% guardrail accuracies, respectively. Project page: https://guardagent.github.io/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。