arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13653cs.AIcs.CLcs.CVcs.HC

通过代理-Q估计和分步策略优化构建自主GUI导航

Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization

National Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);Ovis Team, Alibaba Group(阿里团队,阿里巴巴集团)

Yibo Wang, Guangda Huzhang, Yuwei Hu, Yu Xia, Shiyin Lu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

AI总结:

本文提出基于代理-Q估计和分步策略优化的GUI自主导航框架,通过强化学习提升GUI交互能力,实验证明其在导航和基准测试中表现优异。

摘要:

近年来,多模态大语言模型(MLLMs)的进步显著推动了图形用户界面(GUI)自主代理的发展。然而,在现实应用中,GUI代理常面临非平稳环境,导致数据整理和策略优化的计算成本较高。本文介绍了一种新的以MLLM为中心的GUI代理框架,包含两个组成部分:代理-Q估计和分步策略优化。前者旨在优化一个能够生成分步值的Q模型,以评估给定动作对任务完成的贡献。后者通过从状态-动作轨迹中提取分步样本作为输入,利用我们的代理-Q模型通过强化学习优化策略。值得注意的是(i)所有状态-动作轨迹均由策略本身生成,因此数据收集成本可控;(ii)策略更新与环境解耦,确保了稳定且高效的优化。实证评估显示,我们的框架赋予Ovis2.5-9B强大的GUI交互能力,在GUI导航和基础基准测试中取得显著成绩,甚至超越了规模更大的竞争对手。

英文摘要:

Recent advances in Multimodal Large Language Models (MLLMs) have substantially driven the progress of autonomous agents for Graphical User Interface (GUI). Nevertheless, in real-world applications, GUI agents are often faced with non-stationary environments, leading to high computational costs for data curation and policy optimization. In this report, we introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization. The former one aims to optimize a Q-model that can generate step-wise values to evaluate the contribution of a given action to task completion. The latter one takes step-wise samples from the state-action trajectory as inputs, and optimizes the policy via reinforcement learning with our agentic-Q model. It should be noticed that (i) all state-action trajectories are produced by the policy itself, so that the data collection costs are manageable; (ii) the policy update is decoupled from the environment, ensuring stable and efficient optimization. Empirical evaluations show that our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks and even surpassing contenders with larger scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑