ScreenAgent：一种视觉语言模型驱动的计算机控制智能体
ScreenAgent: A Vision Language Model-driven Computer Control Agent
AI总结：
本文提出ScreenAgent，一种基于视觉语言模型的计算机控制智能体，通过规划、执行、反思流程操作真实GUI，其性能媲美GPT-4V且UI定位更精准，并开源数据集与代码。
摘要：
现有的大语言模型（LLM）可以调用各种工具和API来完成复杂任务。计算机作为最强大且通用的工具，有可能被训练好的LLM智能体直接控制。借助计算机，我们有望构建一个更通用的智能体，以协助人类完成各种日常数字化工作。在本文中，我们构建了一个环境，使视觉语言模型（VLM）智能体能够与真实计算机屏幕进行交互。在该环境中，智能体可以观察屏幕截图，并通过输出鼠标和键盘动作来操作图形用户界面（GUI）。我们还设计了一个自动化控制流程，包括规划、执行和反思三个阶段，引导智能体持续与环境交互并完成多步任务。此外，我们构建了ScreenAgent数据集，该数据集收集了完成各种日常计算机任务时的屏幕截图和动作序列。最后，我们训练了一个模型ScreenAgent，其计算机控制能力可与GPT-4V相媲美，并展现出更精确的UI定位能力。我们的尝试可能激发进一步研究构建通用LLM智能体的兴趣。代码可在\url{https://github.com/niuzaisheng/ScreenAgent}获取。
英文摘要：
Existing Large Language Models (LLM) can invoke a variety of tools and APIs to complete complex tasks. The computer, as the most powerful and universal tool, could potentially be controlled directly by a trained LLM agent. Powered by the computer, we can hopefully build a more generalized agent to assist humans in various daily digital works. In this paper, we construct an environment for a Vision Language Model (VLM) agent to interact with a real computer screen. Within this environment, the agent can observe screenshots and manipulate the Graphics User Interface (GUI) by outputting mouse and keyboard actions. We also design an automated control pipeline that includes planning, acting, and reflecting phases, guiding the agent to continuously interact with the environment and complete multi-step tasks. Additionally, we construct the ScreenAgent Dataset, which collects screenshots and action sequences when completing a variety of daily computer tasks. Finally, we trained a model, ScreenAgent, which achieved computer control capabilities comparable to GPT-4V and demonstrated more precise UI positioning capabilities. Our attempts could inspire further research on building a generalist LLM agent. The code is available at \url{https://github.com/niuzaisheng/ScreenAgent}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
