AMEX:面向移动端 GUI 智能体的 Android 多标注 Expo 数据集
AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
AI总结:
为推进移动端 AI 智能体研究,本文提出大规模多层级标注数据集 AMEX,包含超 104K 截图及元素定位、功能描述和操作链三层标注,并微调 SPHINX Agent 基准验证了其有效性。
摘要:
AI 智能体因其感知环境、理解任务并自主实现目标的能力而受到越来越多的关注。为了推进移动场景下 AI 智能体的研究,我们引入了 Android Multi-annotation EXpo (AMEX),这是一个专为通用移动 GUI 控制智能体设计的综合性大规模数据集,这些智能体能够通过直接与移动设备上的图形用户界面 (GUI) 交互来完成任务。AMEX 包含来自流行移动应用程序的超过 104K 张高分辨率屏幕截图,并进行了多层级标注。与现有的 GUI 相关数据集(例如 Rico、AitW 等)不同,AMEX 包含三个层级的标注:GUI 交互式元素定位、GUI 屏幕及元素功能描述,以及带有逐步 GUI 操作链的复杂自然语言指令。我们从更具指导性和详细的角度开发了该数据集,补充了现有数据集的常规设置。此外,我们微调了基准模型 SPHINX Agent,并证明了 AMEX 的有效性。该项目可在 https://yxchai.com/AMEX/ 获取。
英文摘要:
AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。