Plover：通过以计划为中心的交互来操控图形用户界面代理
Plover: Steering GUI Agents through Plan-Centric Interaction
- University of California, Davis（加利福尼亚大学戴维斯分校）
- Bosch Research North America（博世北美研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
研究针对现实中GUI自动化难题，提出以计划为中心的Plover系统，通过规划器-执行器架构，支持多方式监督与修正，经形成性研究和评估表明，该系统能让GUI自动化更透明、可控与可适应，利于修复故障。
AI中文摘要：
在现实世界环境中，图形用户界面（GUI）自动化颇具挑战，动态布局、意外对话框和不断变化的界面状态会使自主代理偏离用户意图。近期基于视觉的多模态代理虽通过直接操作截图和自然语言指令提高了灵活性，但规划和适应常处于内部，限制了用户检查、监督或纠正系统行为的能力。我们提出了Plover，这是一个以计划为中心的基于视觉的GUI自动化系统，它将任务计划和重新规划作为持久、可检查和可修订的工件外化。通过规划器-执行器架构，Plover支持对不断演变的执行进行明确监督，通过可编辑计划进行局部纠正、自然语言指导以及基于截图的干预，同时在修复过程中保留先前进度。一项有六名参与者的形成性研究为交互设计提供了参考。然后我们通过基准故障案例修复和基于场景的工作流程分析对Plover进行评估。结果表明，当计划可见且干预局部化时，许多自主GUI代理故障在结构上是可修复的，并且明确的重新规划有助于使GUI自动化更透明、可控和可适应。
英文摘要：
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.
