arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04296cs.AI
收藏

饱和陷阱与干预时机的主观性:为什么基于情感的触发器和LLM评判者无法在自主智能体上把握干预时机

The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents

manvendramodgil.ai

Manvendra Modgil

AI总结:

本研究通过18维情感动力学引擎HEART诊断自主智能体干预时机问题,发现状态饱和陷阱、LLM评判者的能力与上下文门槛,以及人类标注者之间极低的干预时机一致性,表明干预时机是一个低可靠性构念。

摘要:

随着自主AI智能体从对话系统转向长周期软件执行,决定何时中断智能体的运行时安全层变得至关重要。我们使用一个连续的18维情感动力学引擎(HEART)作为诊断探针,研究了这一时机问题,评估了四种干预触发家族——绝对状态阈值、复合状态-动作模式、正则推理特征提取和零样本LLM作为评判者——针对SWE-bench-Verified调试轨迹上人工标注的干预点。我们报告了三个发现。首先,状态饱和陷阱:智能体在持续困难下没有恢复信号,因此建模的挫折感迅速越过阈值并保持最大值,将基于状态阈值的触发器从时刻检测器转变为近乎恒定的指示器,在五个轨迹中触发39-83%的动作。其次,LLM评判者的能力和上下文底线:小模型(gpt-5.4-mini)从不触发,而前沿和跨供应商模型只有在完整轨迹上下文下才能逃脱零触发底线,即使如此,F1值也仅为0.17-0.40,成本高达90倍。第三,最重要的是,监督目标在人类之间不可复现:三名训练有素的标注者使用同一评分标准对一条56动作轨迹进行标注,在干预位置上的一致性仅略高于偶然(位置Krippendorff's alpha = +0.047;最佳成对Cohen's kappa = +0.349),而在干预类型上完全不一致(暂停退化;澄清低于偶然;仅反思alpha = +0.226)。我们得出结论,干预时机是一个低可靠性构念,使得单标注者F1不适合作为优化目标。我们的贡献是跨人类评分者间信度、四种检测器架构、跨模型LLM评判者扫描以及复现的饱和效应,共同绘制了这一问题图谱,而非任何单一检测器的准确性。

英文摘要:

As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential. We study this timing problem using a continuous 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe, evaluating four intervention trigger families - absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge - against human-annotated intervention points on SWE-bench-Verified debugging traces. We report three findings. First, a State Saturation Trap: modeled frustration quickly crosses the threshold and stays at its maximum (v2 erratum: the observer could not register recovery; decay never ran), converting threshold-on-state triggers from moment detectors into near-constant indicators that fire on 39-83% of actions across five trajectories. Second, a capability-and-context floor for LLM judges: a small model(gpt-5.4-mini) never fires, while frontier and cross-vendor models escape the zero-firing floor only with full-trajectory context, reaching only F1 0.17-0.40 at up to 90x the cost. Third, and most importantly, the supervised target is not reproducible among humans: three trained annotators using the same rubric on a 56-action trajectory agree on where to intervene only slightly above chance (location Krippendorff's alpha = +0.047; best pairwise Cohen's kappa = +0.349) and not at all on intervention type (pause degenerate; clarify below chance; reflect only alpha = +0.226). We conclude that intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target. Our contribution is the joint mapping of this problem across human inter-rater reliability, four detector architectures, a cross-model LLM-judge sweep, and a reproduced saturation effect, rather than any single detector's accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑