arXiv 2407.009932024-07-01cs.AIcs.CL

Mobile-Bench：一个用于评估基于大语言模型的移动智能体的评测基准

Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents

XiaoMi AI Lab（小米AI实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of Electronic Science and Technology of China（电子科技大学）

Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, Shuo Shang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Mobile-Bench基准，通过融合API操作、划分SAST/SAMT/MAMT三级任务并引入CheckPoint指标，系统评估基于LLM的移动智能体规划与多APP协作能力。

摘要：

随着大语言模型（LLMs）的显著进步，基于LLM的智能体已成为人机交互领域的研究热点。然而，目前针对基于LLM的移动智能体的评测基准十分稀缺。对此类智能体进行基准评测通常面临三个主要挑战：（1）仅依赖UI操作的效率低下对任务评估造成了限制。（2）单一应用内的具体指令不足以评估LLM移动智能体在多维推理与决策方面的能力。（3）现有评测指标不足以准确评估连续动作的过程。为此，我们提出了Mobile-Bench，这是一个用于评估基于LLM的移动智能体能力的新型评测基准。首先，我们通过整合103个收集到的API来扩展传统的UI操作，以加速任务完成的效率。随后，我们结合真实用户查询与LLM的增强来收集评测数据。为了更好地评估移动智能体不同层次的规划能力，我们的数据被划分为三个不同的组别：SAST、SAMT和MAMT，分别反映不同的任务复杂度水平。Mobile-Bench包含832条数据条目，其中超过200个任务专门用于评估多APP协作场景。此外，我们引入了一种更准确的评测指标，名为CheckPoint，用于评估基于LLM的移动智能体在其规划与推理步骤中是否到达关键节点。

英文摘要：

With the remarkable advancements of large language models (LLMs), LLM-based agents have become a research hotspot in human-computer interaction. However, there is a scarcity of benchmarks available for LLM-based mobile agents. Benchmarking these agents generally faces three main challenges: (1) The inefficiency of UI-only operations imposes limitations to task evaluation. (2) Specific instructions within a singular application lack adequacy for assessing the multi-dimensional reasoning and decision-making capacities of LLM mobile agents. (3) Current evaluation metrics are insufficient to accurately assess the process of sequential actions. To this end, we propose Mobile-Bench, a novel benchmark for evaluating the capabilities of LLM-based mobile agents. First, we expand conventional UI operations by incorporating 103 collected APIs to accelerate the efficiency of task completion. Subsequently, we collect evaluation data by combining real user queries with augmentation from LLMs. To better evaluate different levels of planning capabilities for mobile agents, our data is categorized into three distinct groups: SAST, SAMT, and MAMT, reflecting varying levels of task complexity. Mobile-Bench comprises 832 data entries, with more than 200 tasks specifically designed to evaluate multi-APP collaboration scenarios. Furthermore, we introduce a more accurate evaluation metric, named CheckPoint, to assess whether LLM-based mobile agents reach essential points during their planning and reasoning steps.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文