DiMo-GUI：通过模态感知视觉推理推进GUI定位的测试时扩展
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
AI总结：
针对GUI定位中视觉元素杂、语言模糊等问题，提出无需训练的DiMo-GUI框架，通过模态分离推理和动态分层区域聚焦优化定位效果，在标准基准上优于基线推理管线。
摘要：
将自然语言查询定位到图形用户界面（GUI）中面临着独特挑战，原因在于视觉元素的多样性、空间布局的杂乱性以及语言的模糊性。本文提出了DiMo-GUI，这是一种无需训练的GUI定位框架，采用两大核心策略：动态视觉定位与模态感知优化。该方法并未将GUI视为单一整体图像，而是将输入拆分为文本元素和图标元素，使模型能借助通用视觉语言模型对各模态独立推理。当预测存在歧义或错误时，DiMo-GUI会以模型初始预测为中心生成候选聚焦区域，动态集中注意力，并逐步放大子区域以优化定位结果。这种分层细化过程无需额外训练或标注，即可消除视觉拥挤布局的歧义。我们在标准GUI定位基准上评估了该方法，结果显示其相较基线推理管线实现了持续提升，证明了模态分离与区域聚焦推理相结合的有效性。
英文摘要：
Grounding natural language queries in graphical user interfaces (GUIs) poses unique challenges due to the diversity of visual elements, spatial clutter, and the ambiguity of language. In this paper, we introduce DiMo-GUI, a training-free framework for GUI grounding that leverages two core strategies: dynamic visual grounding and modality-aware optimization. Instead of treating the GUI as a monolithic image, our method splits the input into textual elements and iconic elements, allowing the model to reason over each modality independently using general-purpose vision-language models. When predictions are ambiguous or incorrect, DiMo-GUI dynamically focuses attention by generating candidate focal regions centered on the model's initial predictions and incrementally zooms into subregions to refine the grounding result. This hierarchical refinement process helps disambiguate visually crowded layouts without the need for additional training or annotations. We evaluate our approach on standard GUI grounding benchmarks and demonstrate consistent improvements over baseline inference pipelines, highlighting the effectiveness of combining modality separation with region-focused reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
