UFO:面向Windows操作系统交互的UI聚焦智能体
UFO: A UI-Focused Agent for Windows OS Interaction
AI总结:
UFO是首个面向Windows操作系统的UI聚焦智能体,利用GPT-Vision和双智能体框架,通过自然语言命令自动完成跨应用任务,在9个应用中验证了其高效性。
摘要:
我们介绍了UFO,一个创新的UI聚焦智能体,旨在满足Windows操作系统上应用程序的用户请求,利用GPT-Vision的能力。UFO采用双智能体框架,细致观察和分析Windows应用程序的图形用户界面(GUI)和控制信息。这使得智能体能够在单个应用程序内及跨应用程序无缝导航和操作,以满足用户请求,即使涉及多个应用程序。该框架包含一个控制交互模块,无需人工干预即可实现动作接地,并支持全自动执行。因此,UFO将繁琐且耗时的过程转变为仅通过自然语言命令即可完成的简单任务。我们在9个流行的Windows应用程序上对UFO进行了测试,涵盖了反映用户日常使用的多种场景。来自定量指标和真实案例研究的结果强调了UFO在满足用户请求方面的卓越有效性。据我们所知,UFO是首个专门针对Windows操作系统环境内任务完成的UI智能体。UFO的开源代码可在https://github.com/microsoft/UFO获取。
英文摘要:
We introduce UFO, an innovative UI-Focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users' daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment. The open-source code for UFO is available on https://github.com/microsoft/UFO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。