Graphical User Interface (GUI) action grounding is a critical step in GUI automation that maps language instructions to actionable elements on GUI screens. Most recent works of GUI action grounding leverage large GUI datasets to fine-tune MLLMs. However, the fine-tuning data always covers limited GUI environments, and we find the performance of the resulting model deteriorates in novel environments. We argue that the GUI grounding models should be further aligned to the novel environments to reveal their full potential, when the inference is known to involve novel environments, i.e., environments not used during the previous fine-tuning. To realize this, we first propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration and then continuously fine-tune GUI grounding models with the collected data. Our agent leverages a novel Q-value-Incentive In-Context Reinforcement Learning (Q-ICRL) method to optimize exploration efficiency and data quality. Additionally, we introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments and demonstrate the effectiveness of data collected by GUI-Bee in the experiments. Furthermore, we conduct an ablation study to validate the Q-ICRL method in enhancing the efficiency of GUI-Bee. Project page: https://gui-bee.github.io
Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding
Xi’an Jiaotong University(西安交通大学);Princeton University(普林斯顿大学);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学);Michigan State University(密歇根州立大学)
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);VCIP & TMCC & DISSec, College of Computer Science, Nankai University(南开大学计算机学院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院)
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
ServiceNow;Mila - Quebec AI Institute(魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);York University(约克大学);Polytechnique Montréal(蒙特利尔理工学院);McGill University(麦吉尔大学)
Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间安全学院)
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);East China University of Science and Technology(东华大学)
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego(加州大学圣地亚哥分校);ByteDance(字节跳动);University of California, Merced(加州大学默塞德分校);University of Southern California(南加州大学);University at Buffalo(布法罗大学);The University of Queensland(昆士兰大学)
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Shenzhen University of Advanced Technology(深圳理工大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Xiamen University(厦门大学)