利用面向查询的枢轴任务平滑MLLM驱动的GUI智能体的定位与推理
Smoothing Grounding and Reasoning for MLLM-Powered GUI Agents with Query-Oriented Pivot Tasks
AI总结：
提出query inference方法作为GUI定位与推理间的桥梁，通过推断用户查询解决格式差异，在极小数据规模下超越现有grounding技术并提升推理性能。
摘要：
感知增强的预训练，特别是通过grounding技术，被广泛采用以提升图形用户界面（GUI）智能体的性能。然而，在资源受限场景下，面向坐标的grounding与面向动作的推理之间的格式差异限制了grounding在推理任务中的有效性。为应对这一挑战，我们提出一种名为query inference的面向查询的枢轴方法，作为GUI grounding与推理之间的桥梁。通过从截图及其关联元素坐标推断潜在用户查询，query inference提升了对坐标的理解，同时与推理任务更紧密对齐。实验结果表明，在相同训练数据规模下，query inference优于以往的grounding技术。值得注意的是，query inference以不到0.1%的训练数据，实现了与大规模grounding增强的OS-Atlas相当甚至更优的性能。此外，我们探讨了推理格式的影响，并证明将额外语义信息融入输入可进一步提升推理性能。代码公开于https://github.com/ZrW00/GUIPivot。
英文摘要：
Perception-enhanced pre-training, particularly through grounding techniques, is widely adopted to enhance the performance of graphical user interface (GUI) agents. However, in resource-constrained scenarios, the format discrepancy between coordinate-oriented grounding and action-oriented reasoning limits the effectiveness of grounding for reasoning tasks. To address this challenge, we propose a query-oriented pivot approach called query inference, which serves as a bridge between GUI grounding and reasoning. By inferring potential user queries from a screenshot and its associated element coordinates, query inference improves the understanding of coordinates while aligning more closely with reasoning tasks. Experimental results show that query inference outperforms previous grounding techniques under the same training data scale. Notably, query inference achieves comparable or even better performance to large-scale grounding-enhanced OS-Atlas with less than 0.1% of training data. Furthermore, we explore the impact of reasoning formats and demonstrate that integrating additional semantic information into the input further boosts reasoning performance. The code is publicly available at https://github.com/ZrW00/GUIPivot.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
