PSPA-Bench:面向智能手机GUI代理的个性化基准
PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent
AI总结:
本文提出PSPA-Bench,用于评估智能手机GUI代理的个性化能力,通过12855条个性化指令和22款移动应用,揭示现有方法在个性化场景下的不足,并提出改进方向。
摘要:
智能手机GUI代理通过直接操作应用界面执行任务,无需深度系统集成即可实现广泛能力。然而,真实手机使用高度个性化:用户采用多样化的工作流和偏好,挑战代理提供定制化帮助而非通用解决方案。现有GUI代理基准无法充分捕捉个性化维度,因用户特定数据稀疏且缺乏细粒度评估指标。为此,我们提出PSPA-Bench,专门评估智能手机GUI代理的个性化能力。PSPA-Bench包含超过12,855条个性化指令,涵盖10种代表性日常使用场景和22款移动应用,并引入结构感知的过程评估方法,以细粒度衡量代理的个性化能力。通过PSPA-Bench,我们评估了11种最先进的GUI代理。结果表明,当前方法在个性化设置下表现不佳,即使最强代理也仅取得有限成功。我们的分析进一步揭示了三个改进方向:(1)以推理为导向的模型优于通用LLM;(2)感知仍是一个简单但关键的能力;(3)反思和长期记忆机制是提升适应性的关键。这些发现使PSPA-Bench成为系统研究和未来个性化GUI代理进步的基础。
英文摘要:
Smartphone GUI agents execute tasks by operating directly on app interfaces, offering a path to broad capability without deep system integration. However, real-world smartphone use is highly personalized: users adopt diverse workflows and preferences, challenging agents to deliver customized assistance rather than generic solutions. Existing GUI agent benchmarks cannot adequately capture this personalization dimension due to sparse user-specific data and the lack of fine-grained evaluation metrics. To address this gap, we present PSPA-Bench, the benchmark dedicated to evaluating personalization in smartphone GUI agents. PSPA-Bench comprises over 12,855 personalized instructions aligned with real-world user behaviors across 10 representative daily-use scenarios and 22 mobile apps, and introduces a structure-aware process evaluation method that measures agents' personalized capabilities at a fine-grained level. Through PSPA-Bench, we benchmark 11 state-of-the-art GUI agents. Results reveal that current methods perform poorly under personalized settings, with even the strongest agent achieving limited success. Our analysis further highlights three directions for advancing personalized GUI agents: (1) reasoning-oriented models consistently outperform general LLMs, (2) perception remains a simple yet critical capability, and (3) reflection and long-term memory mechanisms are key to improving adaptation. Together, these findings establish PSPA-Bench as a foundation for systematic study and future progress in personalized GUI agents.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。