MobileAgent：通过人机交互与SOP集成增强移动设备控制
MobileAgent: enhancing mobile control via human-machine interaction and SOP integration
AI总结：
MobileAgent通过设计人机交互任务识别敏感信息并集成SOP信息到上下文学习中，在AitW基准上以66.92%的动作成功率实现了无需额外推理成本的最先进移动设备控制性能。
摘要：
以大型语言模型（LLMs）为核心的智能体如今能够为用户自动化移动设备操作。经过微调以学习用户的移动操作后，这些智能体可以在线遵循高层用户指令。它们执行任务，如目标分解、子目标排序和交互式环境探索，直到最终目标实现。然而，在移动操作过程中会出现与个性化用户数据相关的隐私问题，需要用户确认。此外，用户在真实世界中的操作具有探索性，动作数据复杂且冗余，给智能体学习带来了挑战。为解决这些问题，在我们的实际应用中，我们设计了智能体与人类之间的交互任务，以识别敏感信息并对齐个性化用户需求。另外，我们在模型的上下文学习（in-context learning）中集成了标准操作程序（Standard Operating Procedure, SOP）信息，以增强智能体对复杂任务执行的理解。我们的方法在新的设备控制基准AitW上进行了评估，该基准涵盖多步任务中的30K条独特指令，包括应用操作、网页搜索和网页购物。实验结果表明，基于SOP的智能体在不增加额外推理成本的情况下，在LLMs中取得了最先进的性能，总体动作成功率达到66.92%。代码和数据示例可在 https://github.com/alipay/mobile-agent 获取。
英文摘要：
Agents centered around Large Language Models (LLMs) are now capable of automating mobile device operations for users. After fine-tuning to learn a user's mobile operations, these agents can adhere to high-level user instructions online. They execute tasks such as goal decomposition, sequencing of sub-goals, and interactive environmental exploration, until the final objective is achieved. However, privacy concerns related to personalized user data arise during mobile operations, requiring user confirmation. Moreover, users' real-world operations are exploratory, with action data being complex and redundant, posing challenges for agent learning. To address these issues, in our practical application, we have designed interactive tasks between agents and humans to identify sensitive information and align with personalized user needs. Additionally, we integrated Standard Operating Procedure (SOP) information within the model's in-context learning to enhance the agent's comprehension of complex task execution. Our approach is evaluated on the new device control benchmark AitW, which encompasses 30K unique instructions across multi-step tasks, including application operation, web searching, and web shopping. Experimental results show that the SOP-based agent achieves state-of-the-art performance in LLMs without incurring additional inference costs, boasting an overall action success rate of 66.92\%. The code and data examples are available at https://github.com/alipay/mobile-agent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
