面向LLM代理工具使用的可验证安全性
Towards Verifiably Safe Tool Use for LLM Agents
AI总结:
本文提出一种基于STPA的框架,通过形式化规范数据流和工具序列,实现LLM代理的可验证安全性,减少对人工标注的依赖。
摘要:
基于大型语言模型(LLM)的AI代理通过启用数据源、API、搜索引擎、代码沙箱甚至其他代理等工具扩展LLM的能力。尽管这使代理能够执行复杂任务,但LLM可能引发意外的工具交互并引入风险,如泄露敏感数据或覆盖关键记录,这在企业环境中是不可接受的。目前缓解这些风险的方法,如基于模型的安全措施,虽然能提高代理的可靠性,但无法保证系统的安全性。信息流控制(IFC)和时间约束等方法旨在提供保证,但通常需要大量的人工标注。我们提出一个过程,首先通过系统理论过程分析(STPA)识别代理工作流中的危险,推导安全需求,并将其正式化为可执行的数据流和工具序列规范。为此,我们引入了一个能力增强的模型上下文协议(MCP)框架,该框架要求对能力和保密性、信任级别进行结构化标注。这些贡献旨在将基于LLM的代理安全性从随意的可靠性修复转向具有正式保证的主动防护,同时减少对用户确认的依赖,使自主性成为有意识的设计选择。
英文摘要:
Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents to perform complex tasks, LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift LLM-based agent safety from ad hoc reliability fixes to proactive guardrails with formal guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。