不确定性感知的GUI智能体：通过组件推荐和人在回路精炼实现自适应感知
Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement
AI总结：
提出不确定性感知的GUI智能体RecAgent，通过组件推荐和人在回路精炼分别降低感知与决策不确定性，并构建ComplexAction数据集验证其有效性。
摘要：
图形用户界面（GUI）智能体在自动化移动任务方面展现出潜力，但仍面临输入冗余和决策模糊的问题。在本文中，我们提出了RecAgent，一种不确定性感知的智能体，通过自适应感知来解决这些问题。我们区分了GUI导航中的两种不确定性：（1）感知不确定性，由全面屏幕信息带来的输入冗余和噪声引起；（2）决策不确定性，源于模糊的任务和复杂的推理。为减少感知不确定性，RecAgent采用组件推荐机制，识别并聚焦于最相关的UI元素。针对决策不确定性，它使用交互模块在模糊情境下请求用户反馈，从而实现意图感知的决策。这些组件被集成到一个统一框架中，该框架主动降低输入复杂度，并通过人在回路精炼对高不确定性案例做出反应。此外，我们提出了一个名为ComplexAction的数据集，用于评估GUI智能体在复杂场景中执行指定单步操作的成功率。大量实验验证了我们方法的有效性。数据集和代码将在https://github.com/Fanye12/RecAgent上提供。
英文摘要：
Graphical user interface (GUI) agents have shown promise in automating mobile tasks but still struggle with input redundancy and decision ambiguity. In this paper, we present \textbf{RecAgent}, an uncertainty-aware agent that addresses these issues through adaptive perception. We distinguish two types of uncertainty in GUI navigation: (1) perceptual uncertainty, caused by input redundancy and noise from comprehensive screen information, and (2) decision uncertainty, arising from ambiguous tasks and complex reasoning. To reduce perceptual uncertainty, RecAgent employs a component recommendation mechanism that identifies and focuses on the most relevant UI elements. For decision uncertainty, it uses an interactive module to request user feedback in ambiguous situations, enabling intent-aware decisions. These components are integrated into a unified framework that proactively reduces input complexity and reacts to high-uncertainty cases via human-in-the-loop refinement. Additionally, we propose a dataset called \textbf{ComplexAction} to evaluate the success rate of GUI agents in executing specified single-step actions within complex scenarios. Extensive experiments validate the effectiveness of our approach. The dataset and code will be available at https://github.com/Fanye12/RecAgent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
