arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07929cs.IRcs.AI
收藏

相同结果,不同旅程:一种基于轨迹层面的框架,用于比较人类和GUI代理在生产搜索系统中的行为

Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems

Spotify, Sweden(Spotify瑞典);Spotify, Netherlands(Spotify荷兰);Stockholm University, Sweden(斯德哥尔摩大学)

Maria Movin, Claudia Hauff, Aron Henriksson, Panagiotis Papapetrou

AI总结:

本文提出一个基于轨迹的评估框架,比较人类和GUI代理在任务结果、努力、查询构建和界面状态导航中的行为差异,通过实验证明代理行为与人类不同,强调轨迹层面诊断的重要性。

摘要:

由LLM驱动的GUI代理越来越多地用于生产系统中,以自动化工作流并模拟用户进行评估和优化。然而,大多数GUI代理评估侧重于任务成功,并缺乏关于代理是否以人类方式交互的证据。本文提出了一种轨迹层面的评估框架,比较人类和代理在(i)任务结果和努力、(ii)查询构建以及(iii)界面状态导航中的行为。该框架在生产音频流搜索应用中进行了受控研究,其中39名参与者和最先进的GUI代理执行了十项多跳搜索任务。代理在任务成功方面与参与者相当,并生成广泛一致的查询,但遵循系统不同的导航策略:参与者表现出以内容为中心的探索行为,而代理则更偏向于搜索导向和低分支。这些结果表明,结果和查询的一致性并不意味着行为的一致性,这促使在部署GUI代理作为用户代理时进行轨迹层面的诊断。

英文摘要:

LLM-driven GUI agents are increasingly used in production systems to automate workflows and simulate users for evaluation and optimization. Yet most GUI-agent evaluations emphasize task success and provide limited evidence on whether agents interact in human-like ways. We present a trace-level evaluation framework that compares human and agent behavior across (i) task outcome and effort, (ii) query formulation, and (iii) navigation across interface states. We instantiate the framework in a controlled study in a production audio-streaming search application, where 39 participants and a state-of-the-art GUI agent perform ten multi-hop search tasks. The agent achieves task success comparable to participants and generates broadly aligned queries, but follows systematically different navigation strategies: participants exhibit content-centric, exploratory behavior, while the agent is more search-centric and low-branching. These results show that outcome and query alignment do not imply behavioral alignment, motivating trace-level diagnostics when deploying GUI agents as proxies for users in production search systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑