arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.29324cs.CLcs.CV
收藏

STAMP:在可控且可扩展的虚拟环境中训练移动GUI代理的显式记忆

STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments

Tongyi AI Lab, Alibaba Group(通义实验室,阿里巴巴集团);Beijing Jiaotong University(北京交通大学)

Junyang Wang, Haiyang Xu, Xi Zhang, Zhaoqing Zhu, Ming Yan, Jieping Ye, Jitao Sang

AI总结:

提出STAMP框架,通过可控虚拟环境注入确定性记忆变量,生成可验证监督数据并支持在线强化学习,解决移动GUI代理在长时任务中因上下文窗口限制和缺乏显式记忆导致的失败问题。

摘要:

移动GUI代理在即时反应控制方面表现出色,但在需要记忆的现实长时任务中经常失败。这种失败源于有限的上下文窗口与令牌密集的屏幕截图之间的根本冲突。为了节省有限的上下文,代理必须逐步丢弃较旧的视觉历史,永久丢失关键的瞬时信息。此外,现有的以行动为中心的数据集无法教会代理记忆什么或何时显式记忆,并且增强静态真实世界数据成本高昂且缺乏交互验证。为了解决这个问题,我们提出了STAMP,一个通过可控虚拟环境训练移动代理显式记忆的框架,其中确定性记忆变量被程序化地注入到合成任务中,以控制必须记忆的内容、何时编码以及何时检索,从而大规模生成可验证的监督数据,并通过环境驱动的奖励反馈实现在线强化学习。在我们新引入的Memory-World基准测试上评估,得到的Stamp-GUI代理在GUI专用模型中达到了最先进的性能,并在我们的Memory-World基准测试上树立了新的高水位线,展示了卓越的记忆准确性和任务韧性,同时保持了强大的通用移动导航能力。

英文摘要:

Mobile GUI agents excel at immediate reactive control but frequently fail in realistic, long-horizon tasks that require memory. This failure stems from a fundamental conflict between limited context windows and token-heavy screenshots. To save the limited context, agents must progressively discard older visual history, permanently losing crucial transient information. Furthermore, existing action-centric datasets fail to teach agents what or when to explicitly memorize, and augmenting static real-world data is prohibitively expensive and lacks interactive verification. To resolve this, we present STAMP, a framework that trains explicit memory in mobile agents through controllable virtual environments, where deterministic memory variables are programmatically injected into synthesized tasks to control what must be memorized, when it should be encoded, and when it must later be retrieved, thereby producing verifiable supervised data at scale and enabling online reinforcement learning through environment-driven reward feedback. Evaluated on our newly introduced Memory-World benchmark, the resulting Stamp-GUI agent achieves state-of-the-art performance among GUI-specialized models and sets a new high watermark on our Memory-World benchmark, demonstrating exceptional memory accuracy and task resilience while maintaining strong general mobile navigation capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑