arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.02475cs.HC
收藏

Prompt2Task:根据文本提示在智能手机上自动化执行 UI 任务

Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

Tian Huang, Chun Yu, Weinan Shi, Zijian Peng, David Yang, Weiqi Sun, Yuanchun Shi

AI总结:

本文提出 Prompt2Task 系统,利用模拟人类认知的多智能体理解文本提示并自动生成、执行智能手机 UI 任务,实验将成功率从 22.28% 提升至 95.24%。

摘要:

UI 任务自动化通过模拟人类与图形用户界面(GUI)的交互来实现高效的任务执行,而无需修改现有应用程序代码。然而,其更广泛的应用受到限制,因为使用者需要同时具备脚本语言和工作流设计方面的专业知识。为应对这一挑战,我们提出 Prompt2Task,这是一个旨在理解各种与任务相关的文本提示(例如目标、流程),从而生成并执行相应自动化任务的系统。Prompt2Task 集成了一组模拟人类认知功能的智能体,专门负责解释用户意图、管理用于任务生成的外部信息,以及在智能手机上执行操作。这些智能体能够从用户反馈中学习,并基于积累的知识持续提升性能。实验结果表明,成功率从基线的 22.28% 跃升至使用 Prompt2Task 时的 95.24%,并且每个新任务平均只需要 0.69 次用户干预。Prompt2Task 在教程制作、智能辅助和客户服务等领域展现出良好的应用前景。

英文摘要:

UI task automation enables efficient task execution by simulating human interactions with graphical user interfaces (GUIs), without modifying the existing application code. However, its broader adoption is constrained by the need for expertise in both scripting languages and workflow design. To address this challenge, we present Prompt2Task, a system designed to comprehend various task-related textual prompts (e.g., goals, procedures), thereby generating and performing the corresponding automation tasks. Prompt2Task incorporates a suite of intelligent agents that mimic human cognitive functions, specializing in interpreting user intent, managing external information for task generation, and executing operations on smartphones. The agents can learn from user feedback and continuously improve their performance based on the accumulated knowledge. Experimental results indicated a performance jump from a 22.28\% success rate in the baseline to 95.24\% with Prompt2Task, requiring an average of 0.69 user interventions for each new task. Prompt2Task presents promising applications in fields such as tutorial creation, smart assistance, and customer service.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑