SparkUI-Parser: 以鲁棒的定位与解析增强GUI感知能力
SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing
AI总结：
针对现有GUI感知MLLM定位精度低、推理慢且无法全界面解析的问题，提出SparkUI-Parser框架，通过坐标连续建模、改进匈牙利匹配拒绝机制提升性能，还构建了ScreenParse基准，实验显示其在多个基准上优于SOTA方法。
摘要：
现有的用于GUI感知的多模态大语言模型（MLLMs）已取得长足进展，但现有方法仍存在以下挑战：1）它们基于文本自回归机制对离散坐标建模，导致定位精度较低、推理速度较慢。2）它们仅能定位预定义的元素集合，不具备解析整个界面的能力，这限制了其在下游任务中的广泛应用与支持。为解决上述问题，我们提出SparkUI-Parser，这是一种新颖的端到端框架，可同时实现更高的定位精度与整个界面的细粒度解析能力。具体而言，我们没有采用基于概率的离散建模，而是在预训练多模态大语言模型（MLLM）的基础上，新增令牌路由器（token router）和坐标解码器，对坐标进行连续建模。这有效缓解了MLLMs离散输出特性和逐令牌生成过程固有的局限性，从而同时提升了精度与推理速度。为进一步增强鲁棒性，我们引入了基于改进匈牙利匹配算法的拒绝机制，使模型能够识别并拒绝不存在的元素，由此减少误报。此外，我们提出了ScreenParse，这是一个经过严谨构建的基准测试集，用于系统评估GUI模型在不同场景下的结构感知能力。大量实验表明，我们的方法在ScreenSpot、ScreenSpot-v2、CAGUI-Grounding和ScreenParse基准测试上始终优于SOTA方法。相关资源可在https://github.com/antgroup/SparkUI-Parser获取。
英文摘要：
The existing Multimodal Large Language Models (MLLMs) for GUI perception have made great progress. However, the following challenges still exist in prior methods: 1) They model discrete coordinates based on text autoregressive mechanism, which results in lower grounding accuracy and slower inference speed. 2) They can only locate predefined sets of elements and are not capable of parsing the entire interface, which hampers the broad application and support for downstream tasks. To address the above issues, we propose SparkUI-Parser, a novel end-to-end framework where higher localization precision and fine-grained parsing capability of the entire interface are simultaneously achieved. Specifically, instead of using probability-based discrete modeling, we perform continuous modeling of coordinates based on a pre-trained Multimodal Large Language Model (MLLM) with an additional token router and coordinate decoder. This effectively mitigates the limitations inherent in the discrete output characteristics and the token-by-token generation process of MLLMs, consequently boosting both the accuracy and the inference speed. To further enhance robustness, a rejection mechanism based on a modified Hungarian matching algorithm is introduced, which empowers the model to identify and reject non-existent elements, thereby reducing false positives. Moreover, we present ScreenParse, a rigorously constructed benchmark to systematically assess structural perception capabilities of GUI models across diverse scenarios. Extensive experiments demonstrate that our approach consistently outperforms SOTA methods on ScreenSpot, ScreenSpot-v2, CAGUI-Grounding and ScreenParse benchmarks. The resources are available at https://github.com/antgroup/SparkUI-Parser.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
