arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.25864cs.CVcs.AIcs.HC
收藏

GUIDE:一个用于理解和协助用户进行开放性GUI任务的基准测试

GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks

KAIST(韩国科学技术院);Carnegie Mellon University(卡内基梅隆大学);University of Oxford(牛津大学);Konkuk University(建国大学);Google Inc.(谷歌公司);SkillBench

Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim

AI总结:

GUIDE基准测试旨在评估AI模型在开放性GUI任务中感知用户行为、推断意图和提供帮助的能力,通过120名新手用户的10种软件屏幕记录,发现模型在行为状态和帮助预测上准确率仅为44.6%和55.0%,但提供用户上下文可提升帮助预测性能达50.2个百分点。

摘要:

图形用户界面(GUI)代理有潜力帮助用户与复杂软件(如PowerPoint、Photoshop)交互。尽管先前研究主要集中在通过点击和按键自动化用户操作,但这种范式忽略了人类意图,用户希望在探索、迭代和改进想法的同时保持自主性。为超越自动化并实现协作,GUI代理必须理解用户在做什么和为什么。我们引入GUIDE(GUI用户意图检测评估),一个评估AI模型在开放性GUI任务中感知用户行为、推断意图和提供帮助能力的基准测试。GUIDE包含来自120名新手用户在10种软件上127.5小时的屏幕记录,包含语音记录。GUIDE定义了三个任务:(i)行为状态检测,(ii)意图预测,(iii)帮助预测,测试模型识别行为状态、推断目标和决定何时如何帮助的能力。在八个最先进的多模态模型上的评估表明,所有模型都面临挑战,仅在行为状态和帮助预测上达到44.6%和55.0%的准确率。然而,提供用户上下文显著提高了性能,使帮助预测提升高达50.2个百分点,突显了结构化用户理解在有效帮助中的关键作用。我们的数据集可在https://guide-bench.github.io上获取。

英文摘要:

Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑