arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.17439cs.SEcs.AI
收藏

DiagEval: 用于通过GUI代理进行可靠软件评估的轨迹条件诊断

DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents

DeepWisdom(深智科技);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州))

Sirui Hong, Zhijie Liu, Tengfei Li, Wei Tao, Yifan Wu, Chenglin Wu

AI总结:

该研究提出DiagEval,一种基于轨迹的诊断评估协议,用于在GUI代理评估交互式软件后失败时进行诊断。通过重用失败轨迹选择针对性的诊断探针,并将结果聚合为内部归因信号,从而提高准确性,优于基于重试的基线方法。

摘要:

评估LLM生成的交互式软件需要执行和静态分析相结合。关键困难在于正确性是潜在UI状态转换图上的图级可达属性,而GUI评估器只能观察单个执行轨迹。因此,一次失败的执行仅排除了一条实际路径,导致失败归因在评估器侧错误和真实软件缺陷之间变得模糊。我们提出了DiagEval,一种用于后失败GUI代理评估的轨迹条件诊断评估协议。与盲目重试不同,DiagEval重用失败轨迹来选择针对性的诊断探针,并将这些探针的结果聚合为内部归因信号。潜在图视图激发了诊断问题;DiagEval不重建图或估计校准的后验概率。我们在WebDevJudge-Unit和RealDevBench上评估DiagEval,跨多个GUI代理评估器和LLM后端。在误报负例中,DiagEval恢复了最初被错误归因于软件缺陷的45.6-62.1%的失败,优于基于重试的基线方法,相对提升达34.4-160.6%。在完整评估集上,这种恢复将WebDevJudge-Unit的准确性从69.9%提升到78.3%,将RealDevBench的准确性从65.0%提升到81.6%。这些结果表明,可靠的GUI代理评估不仅需要更强的执行,还需要主动的失败诊断来区分评估器侧错误和真正的软件缺陷。我们的代码可在https://github.com/scutGit/DiagEval获取。

英文摘要:

Evaluating LLM-generated interactive software requires execution in addition to static analysis. The key difficulty is that correctness is a graph-level reachable property over latent UI state-transition graphs, whereas a GUI evaluator observes only a single execution trajectory. A failed rollout therefore rules out only one realized path, leaving failure attribution ambiguous between evaluator-side execution error and genuine software defect. We present DiagEval, a trajectory-conditioned diagnostic evaluation protocol for post-failure GUI-agent evaluation of interactive software. Rather than blindly retrying from scratch, DiagEval reuses the failed trajectory to choose targeted diagnostic probes and aggregates their outcomes into an internal attribution signal. The latent-graph view motivates the diagnostic problem; DiagEval does not reconstruct the graph or estimate calibrated posterior probabilities. We evaluate DiagEval on WebDevJudge-Unit and RealDevBench across multiple GUI-agent evaluators and LLM backbones. On false-negative cases, DiagEval recovers 45.6-62.1% of failures that were initially misattributed to software defects, outperforming retry-based baselines with 34.4-160.6% relative gains. On the full evaluation sets, this recovery improves accuracy from 69.9% to 78.3% on WebDevJudge-Unit and from 65.0% to 81.6% on RealDevBench. These results suggest that reliable GUI-agent evaluation requires not only stronger execution, but also active failure diagnosis to disambiguate evaluator-side errors from genuine software defects. Our code is available at https://github.com/scutGit/DiagEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑