UI-AGILE: 通过有效的强化学习和精确的推理时间接地提升GUI代理
UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
AI总结：
UI-AGILE通过改进监督微调过程和推理中的分解接地，提升了GUI代理的接地性能和通用能力，在ScreenSpot-Pro和ScreenSpot-v2基准上实现了最佳表现，地面准确率提升23%。
摘要：
多模态大语言模型（MLLMs）的出现推动了图形用户界面（GUI）代理能力的显著进步。然而，现有的GUI代理训练和推理技术仍然面临推理设计、无效奖励和视觉噪声的困境。为了解决这些问题，我们引入UI-AGILE，以增强训练和推理过程中的GUI代理能力。在训练方面，我们提出了对监督微调（SFT）过程的一系列改进：1）连续奖励函数以激励高精度接地；2）“简单思考”奖励以平衡规划与速度和接地准确性；3）基于裁剪的重采样策略以缓解稀疏奖励问题并提高复杂任务的学习能力。在推理方面，我们提出分解接地与选择，通过将图像分解为更小、可管理的部分，大幅提高高分辨率显示上的接地准确性。实验表明，UI-AGILE在ScreenSpot-Pro和ScreenSpot-v2基准上实现了最先进的接地性能，同时表现出强大的通用代理能力。例如，使用我们训练和推理增强方法，在ScreenSpot-Pro上比最佳基线提升了23%的接地准确率。我们提供代码在https://github.com/KDEGroup/UI-AGILE。
英文摘要：
The emergence of Multimodal Large Language Models (MLLMs) has driven significant advances in Graphical User Interface (GUI) agent capabilities. Nevertheless, existing GUI agent training and inference techniques still suffer from a dilemma for reasoning designs, ineffective reward, and visual noise. To address these issues, we introduce UI-AGILE for enhancing GUI agents at both training and inference. For training, we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process: 1) a continuous reward function to incentivize high-precision grounding; 2) a ``Simple Thinking'' reward to balance planning with speed and grounding accuracy; and 3) a cropping-based resampling strategy to mitigate the sparse reward problem and improve learning on complex tasks. For inference, we present decomposed grounding with selection to dramatically improve grounding accuracy on high-resolution displays by breaking the image into smaller, manageable parts. Experiments show that UI-AGILE achieves the state-of-the-art grounding performance on two benchmarks ScreenSpot-Pro and ScreenSpot-v2 while it also exhibits strong general agent capabilities. For instance, using both our training and inference enhancement methods brings 23\% grounding accuracy improvement over the best baseline on ScreenSpot-Pro. We provide the code in https://github.com/KDEGroup/UI-AGILE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
