MobileFlow: 面向移动GUI智能体的多模态大语言模型
MobileFlow: A Multimodal LLM For Mobile GUI Agent
AI总结:
针对现有GUI智能体的隐私风险、细节丢失及中文界面适配问题,本文提出MobileFlow多模态大模型,通过领域迁移、混合视觉编码器等技术,在任务执行上优于Qwen-VL-Max和GPT-4v并实现业务部署。
摘要:
当前,移动图形用户界面(GUI)的集成在大多数人的日常生活中无处不在。GPT-4v、Qwen-VL-Max等多模态大规模模型的持续演进,显著增强了GUI理解和用户行为分析能力,展现出智能GUI助手的潜力。然而,现有GUI智能体常需通过调用系统API获取页面布局信息,可能带来隐私风险;将GUI(如移动界面)固定到低分辨率可能导致细粒度图像细节丢失;同时,现有面向GUI智能体的多模态大模型对中文GUI界面的理解和决策能力较差,难以应用于大量中文应用。本文介绍MobileFlow,一款专为移动GUI智能体精心打造的多模态大语言模型。它基于开源模型Qwen-VL-Chat向GUI领域迁移,包含约210亿参数,配备新型混合视觉编码器,支持可变分辨率图像输入和多语言GUI。通过融合专家混合(MoE)扩展和开创性对齐训练策略,MobileFlow能够充分解析图像数据并理解GUI交互任务的用户指令。最终,在公共和本文提出的评估指标上,MobileFlow在GUI智能体任务执行方面优于Qwen-VL-Max和GPT-4v,并已成功部署于实际业务场景,证明其实际应用有效性。
英文摘要:
Currently, the integration of mobile Graphical User Interfaces (GUIs) is ubiquitous in most people's daily lives. And the ongoing evolution of multimodal large-scale models, such as GPT-4v, Qwen-VL-Max, has significantly bolstered the capabilities of GUI comprehension and user action analysis, showcasing the potentiality of intelligent GUI assistants. However, current GUI Agents often need to access page layout information through calling system APIs, which may pose privacy risks. Fixing GUI (such as mobile interfaces) to a certain low resolution might result in the loss of fine-grained image details. At the same time, the multimodal large models built for GUI Agents currently have poor understanding and decision-making abilities for Chinese GUI interfaces, making them difficult to apply to a large number of Chinese apps. This paper introduces MobileFlow, a multimodal large language model meticulously crafted for mobile GUI agents. Transforming from the open-source model Qwen-VL-Chat into GUI domain, MobileFlow contains approximately 21 billion parameters and is equipped with novel hybrid visual encoders, making it possible for variable resolutions of image inputs and good support for multilingual GUI. By incorporating Mixture of Experts (MoE) expansions and pioneering alignment training strategies, MobileFlow has the capacity to fully interpret image data and comprehend user instructions for GUI interaction tasks. Finally, MobileFlow outperforms Qwen-VL-Max and GPT-4v in terms of task execution by GUI agents on both public and our proposed evaluation metrics, and has been successfully deployed in real-world business contexts, proving its effectiveness for practical applications.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。