MEGA-GUI：面向GUI元素的多阶段增强定位智能体
MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
AI总结：
针对现有GUI定位方法缺乏模块化、复杂场景表现差的问题，提出多阶段框架MEGA-GUI，通过分阶段定位、双向ROI缩放和语义改写模块，在两个基准上取得超越已有方法的准确率。
摘要：
图形用户界面（GUI）定位是将自然语言指令映射到屏幕坐标的任务，对自主智能体和无障碍技术至关重要。现有系统依赖单体模型或单阶段流水线，缺乏模块化，在视觉杂乱、指令模糊的场景下表现不佳。本文提出MEGA-GUI多阶段框架，将定位拆分为粗粒度感兴趣区域（ROI）选择和细粒度元素定位两个环节，由专门的视觉语言智能体协同调度。MEGA-GUI配备可缓解空间稀释问题的双向ROI缩放算法，以及可降低语义歧义的上下文感知改写智能体。分析发现不同视觉尺度下视觉语言模型的优劣势存在互补性，利用该模块化结构可获得比单体方法更稳定的更高准确率。在视觉元素密集的ScreenSpot-Pro基准上，MEGA-GUI准确率达73.18%；在语义复杂的OSWorld-G基准上准确率达68.63%，均超越此前已公开的最优结果。代码及定位基准工具包（GBT）可在https://github.com/samsungsds-research-papers/mega-gui获取。
英文摘要：
Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
