TRISHUL：面向基于大型VLM的GUI智能体的区域识别与屏幕层级理解
TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents
AI总结：
本文提出无需训练的TRISHUL框架，通过HSP与SEED融合动作定位和GUI指称，增强通用LVLM的整体GUI理解，并在多个基准上取得领先性能。
摘要：
大型视觉语言模型（LVLM）的最新进展，推动了在多种范式下基于LVLM的图形用户界面（GUI）智能体的发展。诸如CogAgent和SeeClick等基于训练的方法，由于依赖特定数据集训练，在跨数据集和跨平台泛化方面存在困难。诸如GPT-4V等通用LVLM采用Set-of-Marks（SoM）进行动作定位，但获取SoM标签需要HTML源代码等元数据，而这些元数据并非在所有平台上都能持续获得。此外，现有方法通常专注于单一GUI任务，而不是实现全面的GUI理解。为解决这些局限，我们提出TRISHUL，这是一种新颖的、无需训练的智能体框架，可增强通用LVLM以实现整体GUI理解。不同于以往工作只关注动作定位（将指令映射到GUI元素）或GUI指称（给定位置描述GUI元素），TRISHUL将二者无缝整合。其核心是，TRISHUL采用Hierarchical Screen Parsing（HSP，分层屏幕解析）和Spatially Enhanced Element Description（SEED，空间增强元素描述）模块，二者协同工作，为GUI元素提供多粒度、空间和语义增强的表示。结果表明，TRISHUL在ScreenSpot、VisualWebBench、AITW和Mind2Web数据集上的动作定位表现优越。此外，在GUI指称方面，TRISHUL在ScreenPR基准上超过ToL智能体，为稳健、适应性强的GUI理解设立了新标准。
英文摘要：
Recent advancements in Large Vision Language Models (LVLMs) have enabled the development of LVLM-based Graphical User Interface (GUI) agents under various paradigms. Training-based approaches, such as CogAgent and SeeClick, struggle with cross-dataset and cross-platform generalization due to their reliance on dataset-specific training. Generalist LVLMs, such as GPT-4V, employ Set-of-Marks (SoM) for action grounding, but obtaining SoM labels requires metadata like HTML source, which is not consistently available across platforms. Moreover, existing methods often specialize in singular GUI tasks rather than achieving comprehensive GUI understanding. To address these limitations, we introduce TRISHUL, a novel, training-free agentic framework that enhances generalist LVLMs for holistic GUI comprehension. Unlike prior works that focus on either action grounding (mapping instructions to GUI elements) or GUI referring (describing GUI elements given a location), TRISHUL seamlessly integrates both. At its core, TRISHUL employs Hierarchical Screen Parsing (HSP) and the Spatially Enhanced Element Description (SEED) module, which work synergistically to provide multi-granular, spatially, and semantically enriched representations of GUI elements. Our results demonstrate TRISHUL's superior performance in action grounding across the ScreenSpot, VisualWebBench, AITW, and Mind2Web datasets. Additionally, for GUI referring, TRISHUL surpasses the ToL agent on the ScreenPR benchmark, setting a new standard for robust and adaptable GUI comprehension.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
