arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.11941cs.CL
收藏

CoCo-Agent:一种用于智能手机GUI自动化的全面认知MLLM智能体

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(上海市Web3可信数据流通与治理重点实验室)

Xinbei Ma, Zhuosheng Zhang, Hai Zhao

AI总结:

提出CoCo-Agent智能体,通过全面环境感知和条件动作预测方法,系统性提升智能手机GUI自动化性能,在AITW和META-GUI基准上达到最先进水平。

摘要:

多模态大语言模型(MLLMs)作为类人自主语言智能体与现实环境交互,尤其在图形用户界面(GUI)自动化方面,展现出巨大潜力。然而,这些GUI智能体需要全面的认知能力,包括详尽的感知和可靠的动作响应。我们提出了一种全面认知LLM智能体CoCo-Agent,包含两种新方法:全面环境感知(CEP)和条件动作预测(CAP),以系统性提升GUI自动化性能。首先,CEP通过不同方面和粒度促进GUI感知,包括视觉通道的截图和补充详细布局,以及文本通道的历史动作。其次,CAP将动作预测分解为子问题:动作类型预测和以动作类型为条件的动作目标预测。凭借我们的技术设计,我们的智能体在AITW和META-GUI基准上实现了新的最先进性能,在真实场景中展现出有前景的能力。代码可在https://github.com/xbmxb/CoCo-Agent获取。

英文摘要:

Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation. However, those GUI agents require comprehensive cognition ability including exhaustive perception and reliable action response. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel. Second, CAP decomposes the action prediction into sub-problems: action type prediction and action target conditioned on the action type. With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https://github.com/xbmxb/CoCo-Agent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑