AutoDroid-V2：通过代码生成提升基于SLM的GUI智能体
AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation
AI总结：
AutoDroid-V2将移动UI任务自动化转化为代码生成问题，通过自动构建应用API文档和任务样本，使设备端小型语言模型能够生成精确脚本，在提升成功率的同时降低延迟和令牌消耗。
摘要：
大型语言模型（LLMs）为移动UI智能体这一长期研究领域带来了令人振奋的新进展，该领域旨在通过移动UI交互完成任意自然语言任务。然而，现有UI智能体通常依赖难以在终端用户设备上本地部署的强大大型语言模型，这引发了关于用户隐私和集中式服务成本的巨大担忧。受近期小型语言模型（SLMs）卓越编码能力的启发，我们提出将UI任务自动化问题转化为代码生成问题，该问题可由设备端SLM有效解决，并通过设备端代码解释器高效执行。与可通过公共数据集进行大规模预训练的普通编码任务不同，生成UI自动化代码具有挑战性，因为目标应用具有多样性、复杂性和可变性。因此，我们采用以文档为中心的方法，自动为每个应用构建细粒度API文档，并基于该文档生成多样化的任务样本。通过使用合成文档和任务样本引导智能体，它学会生成精确高效的脚本来完成未见过的任务。基于与最先进的移动UI智能体的详细比较，我们的方法有效提升了移动任务自动化水平，具有显著更高的成功率和更低的延迟/令牌消耗。代码已在 https://github.com/MobileLLM/AutoDroid-V2 开源。
英文摘要：
Large language models (LLMs) have brought exciting new advances to mobile UI agents, a long-standing research field that aims to complete arbitrary natural language tasks through mobile UI interactions. However, existing UI agents usually demand powerful large language models that are difficult to be deployed locally on end-users' devices, raising huge concerns about user privacy and centralized serving cost. Inspired by the remarkable coding abilities of recent small language models (SLMs), we propose to convert the UI task automation problem to a code generation problem, which can be effectively solved by an on-device SLM and efficiently executed with an on-device code interpreter. Unlike normal coding tasks that can be extensively pre-trained with public datasets, generating UI automation code is challenging due to the diversity, complexity, and variability of target apps. Therefore, we adopt a document-centered approach that automatically builds fine-grained API documentation for each app and generates diverse task samples based on this documentation. By guiding the agent with the synthetic documents and task samples, it learns to generate precise and efficient scripts to complete unseen tasks. Based on detailed comparisons with state-of-the-art mobile UI agents, our approach effectively improves the mobile task automation with significantly higher success rates and lower latency/token consumption. Code is open-sourced at https://github.com/MobileLLM/AutoDroid-V2.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
