通过迭代缩小改进GUI定位
Improved GUI Grounding via Iterative Narrowing
AI总结：
针对通用和微调VLM在GUI定位上的不足，提出基于迭代缩小的视觉提示框架，在综合基准上显著提升定位性能。
摘要：
图形用户界面（GUI）定位在增强视觉语言模型（VLM）智能体的能力方面发挥着至关重要的作用。尽管通用视觉语言模型（如GPT-4V）在各种任务中表现出强大的性能，但它们在GUI定位方面的熟练程度仍不尽如人意。近期研究专注于针对零样本GUI定位对这些模型进行微调，相较于基线性能取得了显著改进。我们提出了一种视觉提示框架，采用迭代缩小机制，以进一步提升通用模型和微调模型在GUI定位中的性能。在评估中，我们在一个涵盖多种UI平台的综合基准上测试了我们的方法，并提供了复现我们结果的代码。
英文摘要：
Graphical User Interface (GUI) grounding plays a crucial role in enhancing the capabilities of Vision-Language Model (VLM) agents. While general VLMs, such as GPT-4V, demonstrate strong performance across various tasks, their proficiency in GUI grounding remains suboptimal. Recent studies have focused on fine-tuning these models specifically for zero-shot GUI grounding, yielding significant improvements over baseline performance. We introduce a visual prompting framework that employs an iterative narrowing mechanism to further improve the performance of both general and fine-tuned models in GUI grounding. For evaluation, we tested our method on a comprehensive benchmark comprising various UI platforms and provided the code to reproduce our results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
