发表机构
Techtouch, Inc.; Institute of Science Tokyo（Techtouch 公司; 东京科学大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过人工审查WebArena-Lite任务，揭示仅依赖最终分数的评估缺陷，并提出记忆与支持机制及指南文本以提升评估准确性。
AI 中文摘要
Web智能体是大语言模型的重要应用，然而其评估通常依赖于仅检查最终结果的基于规则或语言模型的评估器。对任务完成情况的人工验证以及对失败轨迹的详细分析仍然有限。我们基于GPT 5.5和未训练的Qwen3.5 9B模型构建的六种评估条件下，审计了全部165个WebArena Lite任务。该审计保留原始分数，纠正自动评估器产生的假阴性，识别首个后果性错误，并检查轨迹中的进展。我们还研究了一种记忆与分析支持机制（MASM），该机制维护显式执行状态，以及提供任务相关程序性指导的指南文本。在四种GPT 5.5设置中，人工审查恢复了评估器遗漏的5.45至8.49个百分点的成功率。在25步预算下，指南文本将使用MASM的修正成功率从34.55%提升至38.18%。在未训练的Qwen3.5 9B模型上，MASM将评估器分数从13.90%提升至18.80%。对102条失败的GPT 5.5轨迹的审查揭示了频繁的滚动循环、未完成的探索、过早的答案、无效操作以及不完整的表单工作流。步骤级证据进一步表明，大量的早期进展可能与最终失败共存。这些结果表明，仅凭最终分数无法完整描述Web智能体行为，并促使采用基于人工的、轨迹感知的验证。
英文摘要
Web agents are an important application of large language models, yet their evaluation often depends on rule based or language model evaluators that inspect only the final outcome. Human verification of task completion and detailed analysis of failed trajectories remain limited. We audit all 165 WebArena Lite tasks under six evaluation conditions built from GPT 5.5 and an untrained Qwen3.5 9B model. The audit retains the original score, corrects false negatives from the automatic evaluator, identifies the first consequential error, and examines progress across the trajectory. We also study a Memory and Analysis Support Mechanism (MASM), which maintains explicit execution state, and Guide Text, which provides task relevant procedural guidance. Across four GPT 5.5 settings, human review recovers 5.45 to 8.49 percentage points of success missed by the evaluator. With a 25 step budget, Guide Text raises corrected success with MASM from 34.55% to 38.18%. On the untrained Qwen3.5 9B model, MASM raises the evaluator score from 13.90% to 18.80%. Review of 102 failed GPT 5.5 trajectories reveals frequent scrolling loops, unfinished exploration, premature answers, invalid actions, and incomplete form workflows. Step level evidence further shows that substantial early progress can coexist with a final failure. These results show why final scores alone provide an incomplete account of web agent behavior and motivate human grounded, trajectory aware verification.
Comments13 pages, 1 figure, 10 tables. Accepted as a poster at the NeurIPS 2026 Workshop "Who Verifies the Agents? Toward Reliable Agent Development"
