LatentRiskBench:评估企业智能体中依赖状态的风险
LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents
AI总结:
本研究推出LatentRiskBench基准,针对企业智能体的依赖状态安全风险,通过配对轨迹、引入残余风险指标,发现状态层面评估检测违规的准确率远高于动作层面,为企业智能体提供更实际的安全度量。
摘要:
企业智能体面临的安全风险无法仅从单个动作中判定。现有安全基准大多评估动作局部风险,忽略了那些安全风险取决于工作流状态的故障。在这类依赖状态的风险中,同一动作在一个情境中安全,在另一个情境中却可能违反策略。因此,我们推出LatentRiskBench,用于评估企业场景中这类依赖状态的风险。该基准将风险轨迹与良性轨迹配对,二者共享相同的终端动作,但在与策略相关的潜在状态上存在差异,要求模型重构工作流上下文,而非依赖动作层面的线索。我们还引入了用于依赖状态风险评估的新指标,包括一组残余风险(RR)度量。在该基准上的实验显示,动作层面评估仅能检测到0.09的依赖状态违规,而状态层面评估的检测率达到0.96。现有安全护栏方法大多忽略这类风险,除非明确将其应用于工作流状态。对一个商业部署场景的案例研究进一步表明,该基准中嵌入的安全指标能为企业智能体提供更贴合实际的安全度量。我们的数据集和代码可在该https URL获取。
英文摘要:
Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local risks, overlooking failures whose safety depends on workflow state. In such state-dependent risks, the same action may be safe in one context but violate policy in another. Hence, we introduce LatentRiskBench for evaluating such state-dependent risks in enterprise settings. It pairs risky trajectories with benign counterparts that share the same terminal action but differ in latent policy-relevant states, requiring models to reconstruct workflow context rather than rely on action-level cues. We also introduce new metrics for state-dependent risk evaluation, including a suite of residual risk (RR) measures. Experiments on the benchmark show that action-level evaluation detects only 0.09 of state-dependent violations, whereas state-level evaluation reaches 0.96. Existing guardrail methods largely overlook such risks unless they are explicitly applied to workflow states. Case studies on a commercial deployed scenario further show that the safety metrics embedded in the benchmark provide a more realistic measure of safety for enterprise agents. Our dataset and code are available at https://github.com/JingyueCong/LatentRiskBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。