PrecogUI：通过预认知模拟与经验检索实现主动式GUI智能体
PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval
- University of Chinese Academy of Sciences（中国科学院大学）
- The Hong Kong University of Science and Technology（香港科技大学）
- Harbin Institute of Technology（哈尔滨工业大学）
- The Chinese University of Hong Kong（香港中文大学）
- Shenzhen Loop Area Institute（深圳河套学院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
PrecogUI提出预认知架构，通过主动经验池、模拟执行器和执行控制器，将GUI智能体从反应式转为主动式，在强干扰长时程任务中超越现有方法。
AI中文摘要：
现有的反应式图形用户界面（GUI）智能体在长时程、动态场景中常常失败，因为意外的干扰会引发注意力分散和级联故障。为解决这一问题，我们提出了PrecogUI，一种预认知架构，将范式从反应式执行转变为主动式决策。具体而言，我们设计了一个主动经验池（PEP），它将重复出现的异常和成功模式以“状态-动作-结果”三元组的形式缓存在双记忆存储库中。此外，我们引入了一个主动模拟执行器（PSE），它学习在给定候选动作的情况下预测下一个符号化UI布局，从而实现早期异常规避，并根据预测的可靠性对候选动作进行排序。最后，一个预认知执行控制器（PEC）融合这些先验知识和预测，优先处理可预见的异常，并通过闭环错误纠正机制确保执行稳健性。为了进行稳健评估，我们开发了AutoTraj，一个自动数据生成引擎，用于构建InterfereBench，一个针对强干扰下长时程任务的基准测试。实验表明，PrecogUI在InterfereBench上超越了最先进的方法，同时在公共基准测试上保持了有竞争力的性能。代码将公开提供。
英文摘要：
Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures. To address this, we propose PrecogUI, a pre-cognitive architecture that shifts the paradigm from reactive execution to proactive decision-making. Specifically, we design a Proactive Experience Pool (PEP), which caches recurring anomaly and success patterns as "state-action-result" tuples in a dual-memory repository. Furthermore, we introduce a Proactive Simulation Executor (PSE) that learns to forecast the next symbolic UI layout given a candidate action, enabling early anomaly avoidance and ranking candidate actions by predicted reliability. Finally, a Pre-cognitive Execution Controller (PEC) fuses these priors and predictions, prioritizes handling of foreseen anomalies, and ensures execution robustness through a closed-loop error correction mechanism. For robust evaluation, we develop AutoTraj, an automatic data-generation engine, to construct InterfereBench, a benchmark for long-horizon tasks with strong disturbances. Experiments demonstrate that PrecogUI surpasses state-of-the-art methods on InterfereBench while maintaining competitive performance on public benchmarks. The code will be publicly available.
