TrajAD:用于可信大语言模型代理的轨迹异常检测
TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agents
AI总结:
TrajAD通过细粒度过程监督训练的专用验证器,解决大语言模型代理轨迹异常检测问题,提升过程可靠性。
摘要:
我们解决了运行时轨迹异常检测问题,这是使大语言模型代理可信的关键能力。当前的安全措施主要集中在静态输入/输出过滤上。然而,我们认为确保大语言模型代理的可靠性需要审计中间执行过程。在本工作中,我们提出了轨迹异常检测的任务。目标不仅仅是检测,而是精确的错误定位。这一能力对于实现高效的回滚和重试至关重要。为此,我们构建了TrajBench,一个通过扰动和完成策略合成的数据库,以覆盖多样的过程异常。使用此基准,我们研究了模型在过程监督方面的能力。我们发现,通用大语言模型,即使在零样本提示下,也难以识别和定位这些异常。这表明,通用能力并不自动转化为过程可靠性。为了解决这一问题,我们提出了TrajAD,一个通过细粒度过程监督训练的专用验证器。我们的方法优于基线,证明了专门监督对于构建可信代理的重要性。
英文摘要:
We address the problem of runtime trajectory anomaly detection, a critical capability for enabling trustworthy LLM agents. Current safety measures predominantly focus on static input/output filtering. However, we argue that ensuring LLM agents reliability requires auditing the intermediate execution process. In this work, we formulate the task of Trajectory Anomaly Detection. The goal is not merely detection, but precise error localization. This capability is essential for enabling efficient rollback-and-retry. To achieve this, we construct TrajBench, a dataset synthesized via a perturb-and-complete strategy to cover diverse procedural anomalies. Using this benchmark, we investigate the capability of models in process supervision. We observe that general-purpose LLMs, even with zero-shot prompting, struggle to identify and localize these anomalies. This reveals that generalized capabilities do not automatically translate to process reliability. To address this, we propose TrajAD, a specialized verifier trained with fine-grained process supervision. Our approach outperforms baselines, demonstrating that specialized supervision is essential for building trustworthy agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。