发表机构
Stony Brook University（石溪大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究推出辅助UI智能体基准NeXUI，评估其导航、解释及用户控制能力，实验发现Gemini-3.5-Flash在该基准中表现不佳，为相关研发提供基础。
AI 中文摘要
现代网页界面对屏幕阅读器的使用越来越不友好，尤其是在页面动态更新或重要结构隐藏在视觉布局之后时。最近的UI智能体可以在这类界面上执行操作；然而，要让辅助智能体真正有用，它们必须表现为让用户知情并掌控局面的协作者，而非仅代表用户执行操作的工具。大多数现有基准主要通过任务完成情况来评判系统，未评估其解释操作或支持用户监督的能力。我们推出NeXUI，这是一个用于辅助智能体的基准，要求智能体在导航界面的同时，用清晰语言为非视觉使用场景解释每一步操作。NeXUI将现实用户目标与检测到的界面状态配对，使智能体能够从视觉上下文和结构信息两方面进行推理。其评估指标包括安全性、效率和任务成功率，同时还会检查解释是否基于界面状态。在实验中，我们发现NeXUI即便对于最先进的基础模型也颇具挑战性，其中Gemini-3.5-Flash的任务成功率仅为44%，且解释评分较差，这使其成为未来研发的有用基础。通过聚焦导航、解释和用户控制，NeXUI为研究能在现代计算环境中支持盲人和视障用户的智能体提供了更清晰的途径。
英文摘要
Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout. Recent UI agents can act on such interfaces; however, for assistive agents to be truly useful, they must behave as collaborators that keep users informed and in control, rather than as tools that simply take actions on users' behalf. Most existing benchmarks judge systems primarily by task completion, without assessing how well they explain their actions or support user oversight. We introduce NeXUI, a benchmark for assistive agents that must navigate interfaces while explaining each step in clear language for nonvisual use. NeXUI pairs realistic user goals with instrumented interface states, enabling agents to reason from both visual context and structural information. Its evaluation measures safety, efficiency, and task success, while also checking whether explanations are grounded in the interface state. In our experiments, we find that NeXUI remains challenging even for state-of-the-art foundation models, with % Gemini-3.5-Flash achieving only a 44\% success rate and poor explanation scores, making it a useful foundation for future research and development. By focusing on navigation, explanation, and user control, NeXUI provides a clearer way to study agents that can support blind and visually impaired users in modern computing environments.
