arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.19191cs.AI
收藏

OS-Themis:一种可扩展的通用GUI奖励批评框架

OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards

University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);CUHK MMLab(香港中文大学多模态实验室);The Hong Kong University of Science and Technology(香港科学大学);National University of Singapore(新加坡国立大学)

Zehao Li, Zhenyu Wu, Yibo Zhao, Bowen Yang, Jingjing Xie, Zhaoyang Liu, Zhoumianze Liu, Kaiming Jin, Jianze Liang, Zonglin Li, Feng Wu, Bowen Zhou, Zun Wang, Zichen Ding

AI总结:

本文提出OS-Themis框架,通过分解轨迹为可验证里程碑并采用审核机制,提升GUI奖励的可扩展性和准确性,实验表明其在AndroidWorld中提升了在线强化学习和轨迹验证的性能。

摘要:

强化学习(RL)有潜力提升随机环境中GUI代理的鲁棒性,但训练对奖励函数质量高度敏感。现有奖励方法难以兼顾可扩展性与性能。为此,我们提出OS-Themis,一种可扩展且准确的多智能体批评框架。不同于单一评判者,OS-Themis将轨迹分解为可验证的里程碑,以隔离决策所需的关键证据,并采用审查机制严格审计证据链后再作出最终裁定。为便于评估,我们进一步引入OmniGUIRewardBench(OGRBench),一个跨平台的GUI结果奖励综合基准,其中所有评估模型在OS-Themis下均取得最佳性能。在AndroidWorld上的广泛实验表明,OS-Themis在支持在线RL训练时提升了10.3%,在轨迹验证和过滤的自训练循环中提升了6.9%,凸显了其推动智能体进化的潜力。

英文摘要:

Reinforcement Learning (RL) has the potential to improve the robustness of GUI agents in stochastic environments, yet training is highly sensitive to the quality of the reward function. Existing reward approaches struggle to achieve both scalability and performance. To address this, we propose OS-Themis, a scalable and accurate multi-agent critic framework. Unlike a single judge, OS-Themis decomposes trajectories into verifiable milestones to isolate critical evidence for decision making and employs a review mechanism to strictly audit the evidence chain before making the final verdict. To facilitate evaluation, we further introduce OmniGUIRewardBench (OGRBench), a holistic cross-platform benchmark for GUI outcome rewards, where all evaluated models achieve their best performance under OS-Themis. Extensive experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training, and a 6.9% gain when used for trajectory validation and filtering in the self-training loop, highlighting its potential to drive agent evolution.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑