arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.23596cs.AI
收藏

Agent-SAMA:状态感知移动助手

Agent-SAMA: State-Aware Mobile Assistant

Linqiang Guo, Wei Liu, Yi Wen Heng, Tse-Hsun, Chen, Yang Wang

AI总结:

针对现有移动GUI智能体反应式执行、缺乏上下文理解与错误恢复能力的问题,提出状态感知多智能体框架Agent-SAMA,通过有限状态机建模应用执行流程,在多类基准上显著提升任务成功率与错误恢复率。

摘要:

移动图形用户界面(GUI)智能体旨在根据用户指令在应用内或跨应用自主完成任务。尽管近期多模态大语言模型(MLLMs)使这类智能体能够解读UI界面并执行操作,但现有智能体本质上仍属于反应式系统。它们仅基于当前UI界面进行推理,缺乏对应用导航流程的结构化表示,这限制了GUI智能体理解执行上下文、检测意外执行结果以及从错误中恢复的能力。我们提出Agent-SAMA,这是一种状态感知多智能体框架,将应用执行过程建模为有限状态机(FSM),把UI界面视为状态、用户操作视为转移。Agent-SAMA实现了四个专用智能体,它们协作实时构建并使用FSM来指导任务规划、执行验证和错误恢复。我们在两类基准上评估Agent-SAMA:跨应用基准(Mobile-Eval-E、SPA-Bench)和以单应用为主的基准(AndroidWorld)。在Mobile-Eval-E上,Agent-SAMA取得了84.0%的任务成功率和71.9%的错误恢复率;在SPA-Bench上,它达到80.0%的任务成功率和66.7%的错误恢复率。与先前方法相比,Agent-SAMA将任务成功率最高提升了12%,错误恢复成功率最高提升了13.8%。在AndroidWorld上,Agent-SAMA取得63.7%的任务成功率,表现优于所有基线方法。我们的结果表明,结构化状态建模能够增强智能体鲁棒性,可作为未来GUI智能体的轻量级、模型无关记忆层。

英文摘要:

Mobile Graphical User Interface (GUI) agents aim to autonomously complete tasks within or across apps based on user instructions. While recent Multimodal Large Language Models (MLLMs) enable these agents to interpret UI screens and perform actions, existing agents remain fundamentally reactive. They reason over the current UI screen but lack a structured representation of the app navigation flow, limiting GUI agents' ability to understand execution context, detect unexpected execution results, and recover from errors. We introduce Agent-SAMA, a state-aware multi-agent framework that models app execution as a Finite State Machine (FSM), treating UI screens as states and user actions as transitions. Agent-SAMA implements four specialized agents that collaboratively construct and use FSMs in real time to guide task planning, execution verification, and recovery. We evaluate Agent-SAMA on two types of benchmarks: cross-app (Mobile-Eval-E, SPA-Bench) and mostly single-app (AndroidWorld). On Mobile-Eval-E, Agent-SAMA achieves an 84.0% success rate and a 71.9% recovery rate. On SPA-Bench, it reaches an 80.0% success rate with a 66.7% recovery rate. Compared to prior methods, Agent-SAMA improves task success by up to 12% and recovery success by 13.8%. On AndroidWorld, Agent-SAMA achieves a 63.7% success rate, outperforming the baselines. Our results demonstrate that structured state modeling enhances robustness and can serve as a lightweight, model-agnostic memory layer for future GUI agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑