arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~

StepReflect:面向移动GUI智能体的结构化UI过渡反射

StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

Linqiang Guo, Wei Liu, Li Gu, Yang Wang, Tse-Hsun, Chen

arXiv 2608.05587首次发表:更新:

AI 中文总结

针对移动GUI智能体长时序执行的动作反射需求,提出StepReflect模型,在离线和在线实验中均优于GPT-5.2相关方案,成本更低且可本地部署。

AI 中文摘要

自主移动GUI智能体需要精准的动作反射以实现可靠的长时序执行。现有方法依赖于每次动作后的开放式多模态推理,该方式成本高昂且与GUI状态过渡的结构化性质不匹配。我们提出StepReflect,其将每一步GUI反射建模为受显式过渡规范和配对视觉证据约束的监督结构化预测。StepReflect通过包含监督微调、师生蒸馏以及基于偏好和奖励的优化的分阶段流程进行训练。在离线场景下,该8B模型在AndroidWorld上达到82.16%的过渡级准确率,在相同结构化输入下比零样本GPT-5.2高出11.83个百分点。在线场景中,在M3A、Agent-SAMA、MAI-UI-8B和Seed-2.0-Pro四个智能体配置里,StepReflect在三个配置中实现了更高的任务成功率,在第四个配置中与GPT-5.2反射智能体仅相差一个成功任务,且在所有四个配置中均降低了基于GPT的反射产生的付费API费用。这些结果表明,StepReflect是一种实用、可本地部署的替代方案,可用于长时序移动GUI智能体中重复使用前沿模型进行反射的方式。

英文摘要

Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions. We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence. StepReflect is trained through a staged pipeline combining supervised fine-tuning, teacher-student distillation, and preference- and reward-based refinement. Offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points under the same structured input. Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth. It also reduces paid API charges relative to GPT-based reflection in all four configurations. These results establish StepReflect as a practical, locally deployable alternative to repeated frontier-model reflection for long-horizon mobile GUI agents.

论文原文

arXiv 摘要页 · PDF 原文 · HTML 原文

↑