arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02455cs.AIcs.CLcs.SE

Drift-Bench: 通过多轮交互诊断LLM代理在输入故障下的合作破裂

Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction

University of Notre Dame（内布拉斯加大学）

Han Bao, Zheyuan Zhang, Pengcheng Jing, Zhengqing Yuan, Kaiwen Shi, Yanfang Ye

备注：
65 pages, 40 figures
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

Drift-Bench通过多轮交互评估LLM代理在输入故障下的合作破裂，采用角色驱动的用户模拟器和Rise评估协议，揭示澄清效果随用户角色和故障类型的变化，推动代理安全评估。

摘要：

随着大语言模型向自主代理过渡，用户输入经常违反合作假设（例如隐含意图、缺失参数、虚假前提或模糊表达），从而产生执行风险，而纯文本评估无法捕捉这些风险。现有基准通常假设指令明确或限制评估为纯文本、单轮澄清，因此无法衡量在基于现实的执行风险下的多轮澄清。我们介绍了Drift-Bench，这是首个评估代理在输入故障下语用学的诊断基准，通过跨面向状态和面向服务的执行环境的多轮澄清进行评估。基于经典沟通理论，Drift-Bench提供了一个合作破裂的统一分类法，并采用以角色驱动的用户模拟器与Rise评估协议。实验显示在这些故障下性能显著下降，澄清效果因用户角色和故障类型而异。MethodName弥合了澄清研究与代理安全评估之间的差距，使系统能够系统地诊断可能导致不安全执行的故障。

英文摘要：

As Large Language Models transition to autonomous agents, user inputs frequently violate cooperative assumptions (e.g., implicit intent, missing parameters, false presuppositions, or ambiguous expressions), creating execution risks that text-only evaluations do not capture. Existing benchmarks typically assume well-specified instructions or restrict evaluation to text-only, single-turn clarification, and thus do not measure multi-turn disambiguation under grounded execution risk. We introduce \textbf{Drift-Bench}, the first diagnostic benchmark that evaluates agentic pragmatics under input faults through multi-turn clarification across state-oriented and service-oriented execution environments. Grounded in classical theories of communication, \textbf{Drift-Bench} provides a unified taxonomy of cooperative breakdowns and employs a persona-driven user simulator with the \textbf{Rise} evaluation protocol. Experiments show substantial performance drops under these faults, with clarification effectiveness varying across user personas and fault types. \MethodName bridges clarification research and agent safety evaluation, enabling systematic diagnosis of failures that can lead to unsafe executions.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑