Act2Intention：通过从GUI操作推断用户意图开发主动移动智能体的基准
Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions
浏览论文内容
中文总结 AI 辅助
本文提出Act2Intention框架及基准数据集，构建主动移动智能体，经实验验证该基准可显著提升智能体意图理解、预测与执行性能，为主动智能体研究提供标准化平台。
中文摘要 AI 辅助
由多模态大语言模型（MLLM）驱动的移动GUI智能体在人机智能领域展现出应用前景，但当前研究主要聚焦于反应式任务执行，缺乏对用户意图的理解-预测-执行完整流程，而用户意图是主动智能体的核心需求。本文提出Act2Intention框架，通过整合用户意图理解、预测与决策执行构建主动移动智能体。首先，通过数据收集与验证生成构建Act2Intention基准数据集（Act2Intention Bench），该数据集涵盖52个应用中的72511个意图及超70万次操作，是首个基于连续意图-操作轨迹评估主动智能体的基准。进一步开发Act2Intention智能体，通过面向主动的意图理解、个性化主动意图预测及经验引导的意图执行实现主动服务。实验结果显示，在相同智能体框架下，对Act2Intention基准数据集进行监督微调后，意图理解、预测、执行任务的准确率分别比未微调的对应模型提升32.0个绝对百分点、10.25个绝对百分点、6.9个SSR百分点。该成果凸显了Act2Intention基准的必要性与价值，它为主动智能体的开发与评估建立了标准化平台，为意图驱动的人机交互研究奠定了基础。
英文摘要
Mobile GUI Agents powered by multimodal large language models (MLLMs) show promise in human-computer intelligence. However, current research primarily focuses on reactive task execution while lacking a comprehensive understanding-prediction-execution process for user intentions, which are the core requirements of active agents. In this paper, we propose the Act2Intention framework that builds an active mobile agent by integrating understanding, predicting user intentions, and executing decisions. First, we construct the Act2Intention Bench through data collection and validated generation, comprising 72,511 intentions and over 700,000 actions across 52 apps, thereby establishing the first benchmark for evaluating proactive agents via continuous intention-action trajectories. We further develop the Act2Intention Agent, achieving proactive services through Proactive-oriented Intention Understanding, Personalized Proactive Intention Prediction, and Experience-guided Intention Execution. Experimental results show that supervised fine-tuning on Act2Intention Bench yields absolute improvements of +32.0 Acc-S, +10.25 Acc-S, and +6.9 SSR points over non-fine-tuned counterparts under the same agent framework for intention understanding, prediction, and execution, respectively. This success underscores the necessity and value of the Act2Intention Bench, which establishes a standardized platform for developing and evaluating proactive agents and consequently paves the way for research on intention-driven human-computer interaction.
发表机构
- Northwestern Polytechnical University（西北工业大学）
- Tsinghua University（清华大学）
机构由 AI 辅助整理，请以论文原文为准。
