KnowU-Bench: 向交互式、主动式和个性化移动代理评估迈进
KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
AI总结：
KnowU-Bench通过Android仿真环境评估个性化移动代理，涵盖42个通用GUI任务、86个个性化任务和64个主动任务，验证代理在交互中获取用户偏好和主动干预的能力，发现前沿模型在模糊指令下表现下降，揭示偏好获取和干预校准的关键瓶颈。
摘要：
个性化移动代理通过推断用户偏好并校准主动帮助具有日常数字助手的潜力，但现有基准无法捕捉其需求。先前工作评估偏好恢复从静态历史或意图预测从固定上下文，既不测试代理能否通过交互获取缺失偏好，也不测试其是否能决定何时干预、寻求同意或在实时GUI环境中保持沉默。我们引入KnowU-Bench，一个基于可重复Android仿真环境的在线基准，涵盖42个通用GUI任务、86个个性化任务和64个主动任务。不同于将用户偏好视为静态上下文的先前工作，KnowU-Bench隐藏用户资料并仅暴露行为日志，迫使真正的偏好推断而非上下文查找。为支持多轮偏好获取，它实例化了一个基于结构化资料的LLM驱动用户模拟器，使能现实的澄清对话和主动同意处理。除了个性化外，KnowU-Bench还提供对完整主动决策链的全面评估，包括基于GUI执行、同意谈判和拒绝后克制，通过结合基于规则的验证与LLM-as-a-Judge评分的混合协议进行评估。我们的实验揭示了显著的退化：在需要用户偏好推断或干预校准的模糊指令下，即使在前沿模型如Claude Sonnet 4.6上，执行显式任务的代理表现也低于50%。核心瓶颈不是GUI导航而是偏好获取和干预校准，揭示了在熟练界面操作与可信赖的个人协助之间的根本差距。
英文摘要：
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
