Chain-of-Ground: 通过迭代推理和参考反馈提升GUI定位
Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedback
AI总结：
Chain-of-Ground通过迭代推理和参考反馈提升GUI定位，实现68.4的准确率，并在现实世界界面中表现优异。
摘要：
GUI定位旨在将自然语言指令与复杂用户界面中的精确区域对齐。先进的多模态大语言模型在视觉GUI定位方面表现出色，但在小目标或视觉相似目标以及现实世界布局中的歧义问题上仍存在困难。这些限制源于定位能力有限以及现有推理潜力的未充分利用。我们提出了Chain of Ground (CoG) 一种训练免费的多步骤定位框架，利用多模态大语言模型进行迭代视觉推理和优化。与直接预测不同，模型逐步反思并调整其假设，从而实现更准确和可解释的定位。我们的方法在ScreenSpot Pro基准测试中达到68.4的准确率，比之前的方法提高了4.8个百分点。为衡量现实世界通用性，我们引入了TPanel UI数据集，包含420个带有模糊和遮挡等视觉失真标记的工业控制面板。在TPanel UI上，Chain of Ground比强大的基线模型Qwen3 VL 235B提高了6.9个百分点，展示了跨现实世界和数字界面的多步骤训练免费定位的有效性。这些结果突显了通过结构化迭代优化来解锁定位潜力的方向，而不是通过额外训练。
英文摘要：
GUI grounding aims to align natural language instructions with precise regions in complex user interfaces. Advanced multimodal large language models show strong ability in visual GUI grounding but still struggle with small or visually similar targets and ambiguity in real world layouts. These limitations arise from limited grounding capacity and from underuse of existing reasoning potential. We present Chain of Ground CoG a training free multi step grounding framework that uses multimodal large language models for iterative visual reasoning and refinement. Instead of direct prediction the model progressively reflects and adjusts its hypotheses leading to more accurate and interpretable localization. Our approach achieves 68.4 accuracy on the ScreenSpot Pro benchmark an improvement of 4.8 points. To measure real world generalization we introduce TPanel UI a dataset of 420 labeled industrial control panels with visual distortions such as blur and masking. On TPanel UI Chain of Ground improves over the strong baseline Qwen3 VL 235B by 6.9 points showing the effectiveness of multi step training free grounding across real world and digital interfaces. These results highlight a direction for unlocking grounding potential through structured iterative refinement instead of additional training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
