GAIR:通过信息联合推理和群体反思实现GUI自动化
GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection
AI总结:
GAIR是一种基于MLLM的GUI自动化代理框架,通过信息联合推理和群体反思整合异质模型能力,提升GUI自动化性能。
摘要:
构建用于GUI自动化任务的AI系统已经吸引了大量研究努力,其中MLLMs被用于处理用户需求并生成操作。然而,GUI自动化包括范围广泛的任务,从文档处理到在线购物,从CAD到视频编辑。不同任务之间的多样性要求GUI自动化中的MLLMs具有异质能力并掌握多维专业知识,这提出了在构建此类模型时的问题。为了解决这一挑战,我们提出了GAIR:通过信息联合推理和群体反思实现GUI自动化,一种基于MLLM的GUI自动化代理框架,旨在整合知识并结合来自异质模型的能力,以构建具有更高性能的GUI自动化代理系统。由于不同的GUI专用MLLMs是在不同的数据集上训练的,因此具有不同的优势,GAIR引入了一个通用的MLLM来联合处理来自多个GUI专用模型的信息,进一步提高代理框架的性能。通用的MLLM还充当决策者,根据之前收集的信息尝试执行合理的操作。当通用模型认为没有足够的信息进行合理的决策时,GAIR会进入群体反思状态,其中通用模型会根据各模型的优缺点提供不同的指令和提示,推动它们收集更具意义和准确性的信息,以支持更深入的推理和决策。我们通过在GUI基准上的广泛实验评估了GAIR的有效性和可靠性。
英文摘要:
Building AI systems for GUI automation task has attracted remarkable research efforts, where MLLMs are leveraged for processing user requirements and give operations. However, GUI automation includes a wide range of tasks, from document processing to online shopping, from CAD to video editing. Diversity between particular tasks requires MLLMs for GUI automation to have heterogeneous capabilities and master multidimensional expertise, raising problems on constructing such a model. To address such challenge, we propose GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection, a novel MLLM-based GUI automation agent framework designed for integrating knowledge and combining capabilities from heterogeneous models to build GUI automation agent systems with higher performance. Since different GUI-specific MLLMs are trained on different dataset and thus have different strengths, GAIR introduced a general-purpose MLLM for jointly processing the information from multiple GUI-specific models, further enhancing performance of the agent framework. The general-purpose MLLM also serves as decision maker, trying to execute a reasonable operation based on previously gathered information. When the general-purpose model thinks that there isn't sufficient information for a reasonable decision, GAIR would transit into group reflection status, where the general-purpose model would provide GUI-specific models with different instructions and hints based on their strengths and weaknesses, driving them to gather information with more significance and accuracy that can support deeper reasoning and decision. We evaluated the effectiveness and reliability of GAIR through extensive experiments on GUI benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。