arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.07413cs.AIcs.CLcs.HCcs.LG
收藏

DigiData:通用移动控制智能体的训练与评估

DigiData: Training and Evaluating General-Purpose Mobile Control Agents

FAIR at Meta(Meta 的 FAIR 研究组);Meta Reality Labs(Meta 现实实验室);University of Southern California(南加州大学)
Yuxuan Sun, Manchen Wang, Shengyi Qian, William R. Wong, Eric Gan, Pierluca D'Oro, Alejandro Castillejo Munoz, Sneha Silwal, Pedro Matias, Nitin Kamra, Satwik K… 展开作者

Yuxuan Sun, Manchen Wang, Shengyi Qian, William R. Wong, Eric Gan, Pierluca D'Oro, Alejandro Castillejo Munoz, Sneha Silwal, Pedro Matias, Nitin Kamra, Satwik Kottur, Nick Raines, Xuanyi Zhao, Joy Chen, Joseph Greer, Andrea Madotto, Allen Bolourchi, James Valori, Kevin Carlberg, Karl Ridgeway, Joseph Tighe

AI总结:

本文提出大规模多模态数据集DigiData及评估基准DigiData-Bench,解决现有数据集目标多样性与复杂度不足问题,改进评估指标,推动通用移动控制智能体发展。

摘要:

能够控制用户界面的AI智能体有望改变人类与数字设备的交互方式。为加速这一变革,两个基础要素至关重要:一是能让智能体实现复杂且与人类相关目标的高质量数据集,二是能让研究人员和从业者快速提升智能体性能的稳健评估方法。本文介绍DigiData,这是一个为训练移动控制智能体设计的大规模、高质量、多样化多模态数据集。与现有从非结构化交互中获取目标的数据集不同,DigiData通过全面探索应用功能精心构建,具有更高的多样性和目标复杂度。此外,我们提出DigiData-Bench基准,用于评估移动控制智能体在真实世界复杂任务中的表现。我们证明常用的步骤准确率指标无法可靠评估移动控制智能体,并为此提出动态评估协议和AI驱动评估作为严格的替代方案。我们的贡献旨在显著推动移动控制智能体的发展,为更直观有效的人机交互铺平道路。

英文摘要:

AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑