arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.05477cs.CL
收藏

不要盲目行动:通过动作-效果验证和自我修正实现鲁棒的GUI自动化

Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

Beijing University of Technology(北京工业大学);Baidu Inc.(百度公司)

Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu, Huanmin Xu, Qiutong Pan, Haiwei Wang

AI总结:

本文提出VeriGUI框架,通过动作-效果验证和自我修正机制,解决GUI自动化中环境噪声导致的失败问题,提升恢复性能。

摘要:

基于视觉-语言模型(VLMs)的自主GUI代理通常假设环境响应确定性,生成动作而不验证先前操作是否成功。在现实世界中,网络延迟、渲染延迟和系统中断会导致未检测到的动作失败、重复无效行为和灾难性错误累积。此外,学习鲁棒恢复策略具有挑战性,因为在线交互成本高且离线数据集缺乏实时反馈。我们提出VeriGUI(基于验证的GUI代理),明确建模噪声环境下的动作结果和恢复。VeriGUI引入Thinking--Verification--Action--Expectation(TVAE)框架以检测失败并指导纠正推理,并采用两阶段训练流水线,结合Robust SFT与合成失败轨迹以及GRPO与不对称验证奖励。我们进一步构建基于AndroidControl的鲁棒性基准以评估失败识别和纠正能力。实验表明,VeriGUI显著减少失败循环,提高恢复成功率,同时保持竞争标准任务性能。

英文摘要:

Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑