Most visual grounding solutions primarily focus on realistic images. However, applications involving synthetic images, such as Graphical User Interfaces (GUIs), remain limited. This restricts the development of autonomous computer vision-powered artificial intelligence (AI) agents for automatic application interaction. Enabling AI to effectively understand and interact with GUIs is crucial to advancing automation in software testing, accessibility, and human-computer interaction. In this work, we explore Instruction Visual Grounding (IVG), a multi-modal approach to object identification within a GUI. More precisely, given a natural language instruction and a GUI screen, IVG locates the coordinates of the element on the screen where the instruction should be executed. We propose two main methods: (1) IVGocr, which combines a Large Language Model (LLM), an object detection model, and an Optical Character Recognition (OCR) module; and (2) IVGdirect, which uses a multimodal architecture for end-to-end grounding. For each method, we introduce a dedicated dataset. In addition, we propose the Central Point Validation (CPV) metric, a relaxed variant of the classical Central Proximity Score (CPS) metric. Our final test dataset is publicly released to support future research.
VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；East China University of Science and Technology（东华大学）
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University（东国大学计算机科学与人工智能系）；Department of Electronics and Electrical Engineering, Dongguk University（东国大学电子与电气工程系）
UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Xiaomi（小米）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Zhejiang University（浙江大学）；Peng Cheng Laboratory（鹏城实验室）
AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions
TMLR Group, Hong Kong Baptist University（香港 Baptist 大学 TMLR 团体）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Sydney AI Centre, The University of Sydney（悉尼大学 AI 中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
BIMgent: Towards Autonomous Building Modeling via Computer-use Agents
Chair of Computing in Civil and Building Engineering, Technical University of Munich, Germany（土木与建筑工程计算系，慕尼黑技术大学）；TUM Georg Nemetschek Institute, Munich, Germany（慕尼黑技术大学Georg Nemetschek研究所）
GUI-GenBench: Evaluating Image Generation Models as Interactive GUI Environments
StepFun；South China University of Technology（南方科技大学）；Peking University（北京大学）；Tsinghua University（清华大学）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；The University of Chicago（芝加哥大学）；Nanyang Technological University（南洋理工大学）
A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?
Carnegie Mellon University（卡内基梅隆大学）；The Chinese University of Hong Kong（香港中文大学）；KAUST（卡塔尔科技大学）；Johns Hopkins University（约翰霍普金斯大学）；Nanyang Technological University（南洋理工大学）；Renmin University of China（中国人民大学）；The Hong Kong University of Science and Technology（香港科学大学）
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Shenzhen University of Advanced Technology（深圳理工大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Xiamen University（厦门大学）
Understanding Visual Saliency in Mobile User Interfaces
Aalto University（阿尔托大学）；Indian Institute Of Technology, Goa（印度理工学院果阿分校）；Nokia Technologies（诺基亚技术公司）；Yildiz Technical University（耶尔德兹技术大学）；Huawei Technologies（华为技术有限公司）
