针对用户点选屏幕内容的 ScreenPR 任务，本文提出具备层次布局树与 ToL grounding 机制的 Tree-of-Lens 智能体，并通过跨平台基准和移动导航任务验证其屏幕理解及错误动作识别能力。
摘要：
图形用户界面（GUI）是我们与数字设备交互的核心，目前已有越来越多工作致力于构建面向各类 GUI 理解任务的模型。然而，这些工作在很大程度上忽略了一项重要的 GUI 指向任务：基于用户指示点的屏幕阅读，我们将其命名为 Screen Point-and-Read（ScreenPR）任务。当前，该任务主要由僵化的无障碍屏幕阅读工具处理，迫切需要由多模态大语言模型（MLLM）进展驱动的新模型。本文提出一种 Tree-of-Lens（ToL）智能体，利用新颖的 ToL grounding 机制来解决 ScreenPR 任务。基于输入点坐标和相应的 GUI 截图，我们的 ToL 智能体构建一棵层次化布局树。基于该树，ToL 智能体不仅能够理解所指示区域的内容，还能清晰表达元素之间的布局和空间关系。此类布局信息对于准确解读屏幕上的信息至关重要，使我们的 ToL 智能体区别于其他屏幕阅读工具。我们还在一个新提出的 ScreenPR 基准上，将 ToL 智能体与其他基线进行了全面评估；该基准包含来自移动端、Web 和操作系统的 GUI。最后同样重要的是，我们在移动 GUI 导航任务上测试了 ToL 智能体，证明了它可用于识别智能体执行轨迹路径上的错误动作。代码和数据：https://screen-point-and-read.github.io
英文摘要：
Graphical User Interfaces (GUIs) are central to our interaction with digital devices and growing efforts have been made to build models for various GUI understanding tasks. However, these efforts largely overlook an important GUI-referring task: screen reading based on user-indicated points, which we name the Screen Point-and-Read (ScreenPR) task. Currently, this task is predominantly handled by rigid accessible screen reading tools, in great need of new models driven by advancements in Multimodal Large Language Models (MLLMs). In this paper, we propose a Tree-of-Lens (ToL) agent, utilizing a novel ToL grounding mechanism, to address the ScreenPR task. Based on the input point coordinate and the corresponding GUI screenshot, our ToL agent constructs a Hierarchical Layout Tree. Based on the tree, our ToL agent not only comprehends the content of the indicated area but also articulates the layout and spatial relationships between elements. Such layout information is crucial for accurately interpreting information on the screen, distinguishing our ToL agent from other screen reading tools. We also thoroughly evaluate the ToL agent against other baselines on a newly proposed ScreenPR benchmark, which includes GUIs from mobile, web, and operating systems. Last but not least, we test the ToL agent on mobile GUI navigation tasks, demonstrating its utility in identifying incorrect actions along the path of agent execution trajectories. Code and data: https://screen-point-and-read.github.io
GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
Beijing University of Posts and Telecommunications（北京邮电大学）；StepFun；Waseda University（早稻田大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents
Beijing Normal University（北京师范大学）；Zhongguancun Academy（中关村学院）；University of Science and Technology of China（中国科学技术大学）；A*STAR；Zhongguancun Institution of Artificial Intelligence（中关村人工智能研究所）
How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
Zhejiang Lab（浙江实验室）；Hangzhou Research and Development Center（杭州研发中心）；China Mobile（中国移动）；Innovation Center of Yangtze River Delta（长江三角洲创新中心）；Zhejiang University（浙江大学）
What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs
Guangming Laboratory（光明实验室）；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；University of Waterloo（滑铁卢大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））
Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
ServiceNow；Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；York University（约克大学）；Polytechnique Montréal（蒙特利尔理工学院）；McGill University（麦吉尔大学）
ScreenAgent: A Vision Language Model-driven Computer Control Agent
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Ministry of Education, China（教育部知识驱动人机智能工程研究中心）
GUI-ARP: Enhancing Grounding with Adaptive Region Perception for GUI Agents
Wuhan University（武汉大学）；Sun Yat-sen University（中山大学）；Alibaba Group（阿里巴巴集团）；East China Normal University（华东师范大学）；University of Electronic Science and Technology of China（电子科技大学）
GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
Yale University（耶鲁大学）；Zhejiang University（浙江大学）；China University of Geosciences（中国地质大学）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Tongji University（同济大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
MIRAGE: Context-Aware Prompt Injection against Mobile GUI Agents via User-Generated Content
Griffith University（格里菲斯大学）；Quantstamp；Nanyang Technological University（南洋理工大学）；Singapore Management University（新加坡管理学院）；University of New South Wales（新南威尔士大学）；Wake Forest University（威克森林大学）；Independent Researcher（独立研究者）
