UI-E2I-Synth：通过大规模指令合成推进GUI定位
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis
AI总结：
针对GUI指令定位面临的数据集不足、标注成本高及现有基准有局限等问题，提出大规模数据合成流水线UI-E2I-Synth和新基准UI-I2E-Bench，显著提升了模型性能，为后续研究提供支持。
摘要：
大视觉语言模型的最新进展正在加速图形用户界面（GUI）智能体的发展，这类智能体利用类人视觉感知能力提升数字设备上的工作效率。与依赖GUI元数据的方法相比，基于视觉的方法适用性更广——前者依赖平台，且易受实现差异的影响。在这一基于视觉的范式中，GUI指令定位（将用户指令映射到给定截图上对应元素的位置）仍是一项关键挑战，尤其是受限于公开训练数据集不足以及手动标注指令数据资源消耗大的问题。本文深入探讨了该任务中尚未被充分研究的挑战，包括元素屏幕占比、元素类型不平衡以及隐式指令问题。为应对这些挑战，我们提出了大规模数据合成流水线UI-E2I-Synth，无需人工标注员，转而使用GPT-4o生成不同复杂度的指令数据集。此外，我们提出了全新的GUI指令定位基准UI-I2E-Bench，该基准纳入多维度标注，旨在解决现有基准的局限性。基于合成数据训练的模型在GUI指令定位任务上取得了更优性能，证明了所提出的数据合成流水线的先进性。所提出的基准结合大量分析，为GUI定位领域的未来研究提供了实用见解。我们将在https://microsoft.github.io/FIVE-UI-Evol/ 发布相关成果。
英文摘要：
Recent advancements in Large Vision-Language Models are accelerating the development of Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities to enhance productivity on digital devices. Compared to approaches predicated on GUI metadata, which are platform-dependent and vulnerable to implementation variations, vision-based approaches offer broader applicability. In this vision-based paradigm, the GUI instruction grounding, which maps user instruction to the location of corresponding element on the given screenshot, remains a critical challenge, particularly due to limited public training dataset and resource-intensive manual instruction data annotation. In this paper, we delve into unexplored challenges in this task including element-to-screen ratio, unbalanced element type, and implicit instruction. To address these challenges, we introduce a large-scale data synthesis pipeline UI-E2I-Synth for generating varying complex instruction datasets using GPT-4o instead of human annotators. Furthermore, we propose a new GUI instruction grounding benchmark UI-I2E-Bench, which is designed to address the limitations of existing benchmarks by incorporating diverse annotation aspects. Our model, trained on the synthesized data, achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline. The proposed benchmark, accompanied by extensive analyses, provides practical insights for future research in GUI grounding. We will release corresponding artifacts at https://microsoft.github.io/FIVE-UI-Evol/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
