Multimodal large language models (MLLMs) are transforming the capabilities of graphical user interface (GUI) agents, facilitating their transition from controlled simulations to complex, real-world applications across various platforms. However, the effectiveness of these agents hinges on the robustness of their grounding capability. Current GUI agents predominantly utilize text-based representations such as HTML or accessibility trees, which, despite their utility, often introduce noise, incompleteness, and increased computational overhead. In this paper, we advocate a human-like embodiment for GUI agents that perceive the environment entirely visually and directly perform pixel-level operations on the GUI. The key is visual grounding models that can accurately map diverse referring expressions of GUI elements to their coordinates on the GUI across different platforms. We show that a simple recipe, which includes web-based synthetic data and slight adaptation of the LLaVA architecture, is surprisingly effective for training such visual grounding models. We collect the largest dataset for GUI visual grounding so far, containing 10M GUI elements and their referring expressions over 1.3M screenshots, and use it to train UGround, a strong universal visual grounding model for GUI agents. Empirical results on six benchmarks spanning three categories (grounding, offline agent, and online agent) show that 1) UGround substantially outperforms existing visual grounding models for GUI agents, by up to 20% absolute, and 2) agents with UGround outperform state-of-the-art agents, despite the fact that existing agents use additional text-based input while ours only uses visual perception. These results provide strong support for the feasibility and promises of GUI agents that navigate the digital world as humans do.
Trifuse: Enhancing Attention-Based GUI Grounding via Multimodal Fusion
College of Computer Science and Technology, National University of Defense Technology（计算机科学与技术学院，国防科技大学）；Intelligent Game and Decision Lab, Academy of Military Sciences（智能游戏与决策实验室，军事科学院）
GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
Wuhan University（武汉大学）；Sun Yat-sen University（中山大学）；Alibaba Group（阿里巴巴集团）；East China Normal University（华东师范大学）；University of Electronic Science and Technology of China（电子科技大学）
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；East China University of Science and Technology（东华大学）
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；University of California, Merced（加州大学默塞德分校）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；The University of Queensland（昆士兰大学）
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Shenzhen University of Advanced Technology（深圳理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Xiamen University（厦门大学）
