迈向GUI代理：用于GUI定位的视觉-语言扩散模型
Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
AI总结：
本文探讨了离散扩散视觉-语言模型在GUI定位中的应用，通过混合掩码策略提升定位精度，并在多个数据集上验证了其有效性。
摘要：
自回归（AR）视觉-语言模型（VLMs）长期以来主导多模态理解和推理以及图形用户界面（GUI）定位。最近，离散扩散视觉-语言模型（DVLMs）在多模态推理中表现出色，提供了双向注意力、并行标记生成和迭代细化。然而，其在GUI定位中的潜力尚未被探索。在本工作中，我们评估离散DVLMs是否能作为AR模型的可行替代方案用于GUI定位。我们调整LLaDA-V以进行单轮动作和边界框预测，将任务框架为从多模态输入生成文本。为更好地捕捉边界框几何结构的层次性，我们提出了一种混合掩码计划，结合线性和确定性掩码，通过在GUI适应的LLaDA-V上训练线性掩码，将定位准确性提高了多达6.1个百分点的步骤成功率（SSR）。在四个涵盖网页、桌面和移动界面的数据集上的评估显示，经过调整的扩散模型在混合掩码下始终优于线性掩码变体，并在有限预训练的情况下与自回归模型竞争。系统消融分析表明，增加扩散步骤、生成长度和块长度会提高准确性，但也增加延迟，准确性在一定扩散步骤数后趋于平稳。通过扩展训练数据以包含多样化的GUI领域，进一步减少延迟约1.3秒，并在多个基准上将定位准确性提高平均20个百分点。这些结果表明，离散DVLMs是GUI定位的有前途的建模框架，并代表了迈向基于扩散的GUI代理的重要一步。
英文摘要：
Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding. We adapt LLaDA-V for single-turn action and bounding-box prediction, framing the task as text generation from multimodal input. To better capture the hierarchical structure of bounding-box geometry, we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points in Step Success Rate (SSR) over the GUI-adapted LLaDA-V trained with linear masking. Evaluations on four datasets spanning web, desktop, and mobile interfaces show that the adapted diffusion model with hybrid masking consistently outperforms the linear-masked variant and performs competitively with autoregressive counterparts despite limited pretraining. Systematic ablations reveal that increasing diffusion steps, generation length, and block length improves accuracy but also increases latency, with accuracy plateauing beyond a certain number of diffusion steps. Expanding the training data with diverse GUI domains further reduces latency by about 1.3 seconds and improves grounding accuracy by an average of 20 points across benchmarks. These results demonstrate that discrete DVLMs are a promising modeling framework for GUI grounding and represent an important step toward diffusion-based GUI agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
