不要开始无法完成的任务:对LLM代理支持状态分拣的反事实审计
Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents
AI总结:
本文通过反事实审计框架评估LLM代理在任务分拣中的能力,发现默认执行过度承诺非完整任务,而通过特定提示条件可显著提升分拣准确性。
摘要:
当前代理评估主要奖励完全指定任务的执行,而近期研究多孤立探讨澄清、能力意识、回避和搜索终止等。本文引入支持状态分拣审计(SSTA-32),通过反事实编辑在四个支持状态间切换:完整(ANSWER)、可澄清(CLARIFY)、支持受阻(REQUEST SUPPORT)和不支持现在(ABSTAIN)。评估前沿模型时,发现默认执行对非完整任务过度承诺(41.7%),而置信度映射避免过度承诺但压缩了三重延迟空间(58.3%)。相反,Action-Only和PSC通过提示暴露类别本体,实现91.7%的延迟准确性。针对性消融实验表明,移除支持充分性维度会降低REQUEST SUPPORT准确性,而移除证据充分性维度会引发对不支持项的系统性过度承诺。由于DPAA在单一上下文窗口内运行,这些结果代表上界能力估计;但结构发现表明,前沿模型具备强潜在分拣能力,需显式类别决策路径来安全激活。
英文摘要:
Current agent evaluations largely reward execution on fully specified tasks, while recent work studies clarification [11, 22, 2], capability awareness [9, 1], abstention [8, 14], and search termination [20, 5] mostly in isolation. This leaves open whether agents can diagnose why a task is blocked before acting. We introduce the Support-State Triage Audit (SSTA-32), a matched-item diagnostic framework in which minimal counterfactual edits flip the same base request across four support states: Complete (ANSWER), Clarifiable (CLARIFY), Support-Blocked (REQUEST SUPPORT), and Unsupported-Now (ABSTAIN). We evaluate a frontier model under four prompting conditions - Direct, Action-Only, Confidence-Only, and a typed Preflight Support Check (PSC) - using Dual-Persona Auto-Auditing (DPAA) with deterministic heuristic scoring. Default execution overcommits heavily on non-complete tasks (41.7% overcommitment rate). Scalar confidence mapping avoids overcommitment but collapses the three-way deferral space (58.3% typed deferral accuracy). Conversely, both Action-Only and PSC achieve 91.7% typed deferral accuracy by surfacing the categorical ontology in the prompt. Targeted ablations confirm that removing the support-sufficiency dimension selectively degrades REQUEST SUPPORT accuracy, while removing the evidence-sufficiency dimension triggers systematic overcommitment on unsupported items. Because DPAA operates within a single context window, these results represent upper-bound capability estimates; nonetheless, the structural findings indicate that frontier models possess strong latent triage capabilities that require explicit categorical decision paths to activate safely.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。