arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.13896cs.CLcs.AIcs.CVcs.LG
收藏

GUI-Bee:通过自主探索将 GUI 动作定位对齐到新环境

GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration

University of California Santa Cruz(加州大学圣克鲁兹分校);Adobe Research(奥多比研究院)

Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, Gang Wu

AI总结:

提出基于MLLM的自主智能体GUI-Bee,通过Q-ICRL方法探索收集高质量环境数据并微调模型,结合NovelScreenSpot基准,有效提升了GUI动作定位在新环境中的性能。

摘要:

图形用户界面(GUI)动作定位是 GUI 自动化中的关键步骤,它将语言指令映射到 GUI 屏幕上的可操作元素。最近大多数 GUI 动作定位的工作利用大型 GUI 数据集来微调 MLLMs。然而,微调数据总是覆盖有限的 GUI 环境,我们发现由此产生的模型在新环境中的性能会下降。我们认为,当已知推理涉及新环境(即先前微调期间未使用的环境)时,GUI 定位模型应进一步对齐到新环境,以展现其全部潜力。为了实现这一点,我们首先提出 GUI-Bee,一个基于 MLLM 的自主智能体,通过探索收集高质量、特定于环境的数据,然后使用收集到的数据持续微调 GUI 定位模型。我们的智能体利用一种新颖的 Q 值激励上下文强化学习(Q-ICRL)方法来优化探索效率和数据质量。此外,我们引入 NovelScreenSpot,一个用于测试数据能多大程度帮助将 GUI 动作定位模型对齐到新环境的基准,并在实验中展示了 GUI-Bee 收集数据的有效性。进一步地,我们进行消融研究以验证 Q-ICRL 方法在提升 GUI-Bee 效率方面的作用。项目页面:https://gui-bee.github.io

英文摘要:

Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee. Project page: https://gui-bee.github.io

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑