arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18429cs.CV
收藏

AndroTMem: 从交互轨迹到长周期GUI代理的锚定记忆

AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

XJTU(西安交通大学);HKUST(GZ)(香港科技大学(广州));HKUST(香港科技大学);CityU(城市大学);UTS(新南威尔士大学);TJU(天津大学);FDU(福建师范大学);SDU(山东大学);CASIA(中国科学院自动化所);SYSU(南方科技大学);NWPU(西北工业大学)
Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghu… 展开作者

Yibo Shi, Jungang Li, Linghao Zhang, Zihao Dongfang, Biao Wu, Sicheng Tao, Yibo Yan, Chenxi Qin, Weiting Liu, Zhixin Lin, Hanqian Li, Yu Huang, Song Dai, Yonghua Hei, Yue Ding, Xiang Li, Shikang Wang, Chengdong Xu, Jingqi Liu, Xueying Ma, Zhiwen Zheng, Xiaofei Zhang, Bincheng Wang, Nichen Yang, Jie Wu, Lihua Tian, Chen Li, Xuming Hu

AI总结:

本文提出AndroTMem框架,通过锚定状态记忆提升长周期GUI代理性能,实验显示其在任务完成率和动作成功率上优于传统方法。

摘要:

长周期GUI代理是实现真实世界部署的关键步骤,但现有交互记忆方法仍存在不足。重放完整交互序列冗余且增加噪声,而摘要常丢失依赖关键信息和可追溯性。我们提出AndroTMem,一种用于长周期Android GUI代理锚定记忆的诊断框架。其核心基准AndroTMem-Bench包含1,069个任务,34,473个交互步骤(平均32.1个/任务,最大65个)。我们评估了代理的TCR(任务完成率),重点关注需要延续关键中间状态的任务;AndroTMem-Bench设计用于强制强步与步因果依赖,使稀疏但必要的中间状态对下游动作至关重要。在开源和闭源GUI代理中,我们观察到一致模式:随着交互序列变长,性能下降主要由任务内记忆失败驱动,而非孤立感知错误或局部动作错误。基于此诊断,我们提出锚定状态记忆(ASM),将交互序列表示为一组因果关联的中间状态锚点,以实现子目标定向检索和属性感知决策。在多个设置和12个评估的GUI代理中,ASM在任务完成率和动作成功率上均优于全序列重放和摘要基线方法,提高了5%-30.16%的TCR和4.93%-24.66%的AMS,表明锚定、结构化的记忆有效缓解了长周期GUI任务中的交互记忆瓶颈。代码、基准和相关资源可在[https://github.com/CVC2233/AndroTMem](https://github.com/CVC2233/AndroTMem)公开获取。

英文摘要:

Long-horizon GUI agents are a key step toward real-world deployment, yet effective interaction memory under prevailing paradigms remains under-explored. Replaying full interaction sequences is redundant and amplifies noise, while summaries often erase dependency-critical information and traceability. We present AndroTMem, a diagnostic framework for anchored memory in long-horizon Android GUI agents. Its core benchmark, AndroTMem-Bench, comprises 1,069 tasks with 34,473 interaction steps (avg. 32.1 per task, max. 65). We evaluate agents with TCR (Task Complete Rate), focusing on tasks whose completion requires carrying forward critical intermediate state; AndroTMem-Bench is designed to enforce strong step-to-step causal dependencies, making sparse yet essential intermediate states decisive for downstream actions and centering interaction memory in evaluation. Across open- and closed-source GUI agents, we observe a consistent pattern: as interaction sequences grow longer, performance drops are driven mainly by within-task memory failures, not isolated perception errors or local action mistakes. Guided by this diagnosis, we propose Anchored State Memory (ASM), which represents interaction sequences as a compact set of causally linked intermediate-state anchors to enable subgoal-targeted retrieval and attribution-aware decision making. Across multiple settings and 12 evaluated GUI agents, ASM consistently outperforms full-sequence replay and summary-based baselines, improving TCR by 5%-30.16% and AMS by 4.93%-24.66%, indicating that anchored, structured memory effectively mitigates the interaction-memory bottleneck in long-horizon GUI tasks. The code, benchmark, and related resources are publicly available at [https://github.com/CVC2233/AndroTMem](https://github.com/CVC2233/AndroTMem).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑