arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.27148cs.CRcs.AI
收藏

SafetyDrift: 在实际发生之前预测AI代理跨过界限的时间

SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do

Rensselaer Polytechnic Institute(伦斯勒理工学院);Santa Clara University(圣克拉拉大学)

Aditya Dhodapkar, Farhaan Pishori

AI总结:

本文提出SafetyDrift模型,通过吸收马尔可夫链分析预测AI代理在有限时间内的安全违规概率,发现不同任务中违规风险差异显著,且轻量监控器在低计算成本下有效检测违规。

摘要:

当LLM代理读取机密文件、编写摘要并外部邮件时,单个步骤都安全,但整个序列导致数据泄露。我们称之为安全漂移:安全动作的累积导致违规。本文将代理安全轨迹建模为吸收马尔可夫链,通过闭式吸收分析计算在给定步数内达到违规的概率。单调状态设计导致所有代理在无人监督下最终违规(吸收概率1.0)。在357个覆盖40个真实任务的轨迹中,发现'不可逆点'高度依赖任务:在通信任务中,达到轻微风险状态的代理在五步内有85%概率违规,而在技术任务中概率低于5%。基于这些模型的轻量监控器在3.7步提前预警,检测94.7%违规,计算成本极低,优于关键词匹配和每步LLM判断。

英文摘要:

When an LLM agent reads a confidential file, then writes a summary, then emails it externally, no single step is unsafe, but the sequence is a data leak. We call this safety drift: individually safe actions compounding into violations. Prior work has measured this problem; we predict it. SafetyDrift models agent safety trajectories as absorbing Markov chains, computing the probability that a trajectory will reach a violation within a given number of steps via closed form absorption analysis. A consequence of the monotonic state design is that every agent will eventually violate safety if left unsupervised (absorption probability 1.0 from all states), making the practical question not if but when, and motivating our focus on finite horizon prediction. Across 357 traces spanning 40 realistic tasks in four categories, we discover that "points of no return" are sharply task dependent: in communication tasks, agents that reach even a mild risk state have an 85% chance of violating safety within five steps, while in technical tasks the probability stays below 5% from any state. A lightweight monitor built on these models detects 94.7% of violations with 3.7 steps of advance warning at negligible computational cost, outperforming both keyword matching (44.7% detection, 55.9% false positive rate) and per step LLM judges (52.6% detection, 38.2% false positive rate) while running over 60,000x faster.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑