ActionEngine: 通过状态机内存实现从响应式到程序化GUI代理的转变
ActionEngine: From Reactive to Programmatic Web Agents via State Machine Memory
AI总结:
ActionEngine通过状态机内存实现从响应式到程序化GUI代理的转变,提升效率和准确性,减少成本和延迟。
摘要:
现有的图形用户界面(GUI)代理通过逐步调用视觉语言模型进行操作——截屏、推理下一步动作、执行它,然后在新页面上重复这一过程,导致成本和延迟随着推理步骤的数量而增加,并且由于没有持久记忆之前访问的页面而限制了准确性。\n我们提出了ActionEngine,一个无需训练的框架,通过一种新的双代理架构实现从响应式执行到程序化规划的转变:一个爬虫代理通过离线探索构建GUI的状态机内存,另一个执行代理利用这种内存来合成完整的可执行Python程序用于在线任务执行。\n为了确保对演变界面的鲁棒性,执行失败会触发基于视觉的重新定位回退,修复失败的动作并更新内存。\n这种设计显著提高了效率和准确性:在WebArena基准的Reddit任务中,我们的代理在平均每次LLM调用的情况下实现了95%的任务成功率,相比最强的视觉-only基线为66%,同时将成本降低了11.8倍,端到端延迟减少了2倍。\n这些组件共同实现了可扩展且可靠的GUI交互,通过结合全局程序化规划、爬虫验证的动作模板、以及节点级执行与局部验证和修复。\n
英文摘要:
Many web agents operate through a reactive execution loop: they observe the current interface, reason about the next action, execute it, and repeat. This design incurs latency and cost that grow with the number of actions, while requiring agents to repeatedly rediscover how the same web application works. We present ActionEngine, a novel architecture that replaces step-by-step reasoning with programmatic execution using reusable knowledge of the application. A Crawling Agent explores the application offline and constructs an updatable state-machine memory that represents its GUI states, the operations available in each state, and the transitions between states. Unlike trajectory memory, this representation stores how the application works rather than solutions to individual tasks. At runtime, an Execution Agent uses this memory to synthesize a complete executable program in a single planning step, which is then executed deterministically without further planning calls. When the interface changes or the memory is incomplete, a reactive fallback repairs the failed action and updates the memory for future tasks. On 655 tasks across four WebArena domains, ActionEngine achieves a 91.2% success rate, outperforming the strongest reactive baseline, Claude Code, by 8.5 percentage points while reducing average task latency by 3.2x and cost by 8x.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。