V-Zen：一种新型多模态大语言模型实现高效的GUI理解与精准定位
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM
AI总结：
V-Zen是一种新型多模态大语言模型，通过双分辨率图像编码器实现高效GUI理解与精准定位，结合GUIDE数据集微调，提升下一步动作预测，为自主计算系统奠定基础。
摘要：
在人工智能研究和应用快速发展的背景下，多模态大语言模型（MLLMs）已成为一股变革性力量，擅长解释和整合来自文本、图像和图形用户界面（GUIs）等多种模态的信息。尽管取得了这些进展，但对图形用户界面的细微交互和理解仍构成重大挑战，限制了现有模型提升自动化水平的潜力。为弥补这一差距，本文提出了V-Zen，一种创新的多模态大语言模型（MLLM），精心设计以革新图形用户界面理解与定位领域。配备双分辨率图像编码器，V-Zen在高效定位和下一步动作预测方面树立了新基准，从而为自操作计算机系统奠定基础。与V-Zen相辅相成的是GUIDE数据集，这是一个包含真实世界GUI元素和基于任务序列的广泛集合，作为专门微调的催化剂。V-Zen与GUIDE的成功整合标志着多模态AI研究新时代的到来，为智能、自主的计算体验打开了大门。本文邀请研究社区加入这一激动人心的旅程，共同塑造GUI自动化的未来。秉持开放科学的精神，我们的代码、数据和模型将公开提供，为具有复杂且精准交互的多模态对话场景铺平道路。
英文摘要：
In the rapidly evolving landscape of AI research and application, Multimodal Large Language Models (MLLMs) have emerged as a transformative force, adept at interpreting and integrating information from diverse modalities such as text, images, and Graphical User Interfaces (GUIs). Despite these advancements, the nuanced interaction and understanding of GUIs pose a significant challenge, limiting the potential of existing models to enhance automation levels. To bridge this gap, this paper presents V-Zen, an innovative Multimodal Large Language Model (MLLM) meticulously crafted to revolutionise the domain of GUI understanding and grounding. Equipped with dual-resolution image encoders, V-Zen establishes new benchmarks in efficient grounding and next-action prediction, thereby laying the groundwork for self-operating computer systems. Complementing V-Zen is the GUIDE dataset, an extensive collection of real-world GUI elements and task-based sequences, serving as a catalyst for specialised fine-tuning. The successful integration of V-Zen and GUIDE marks the dawn of a new era in multimodal AI research, opening the door to intelligent, autonomous computing experiences. This paper extends an invitation to the research community to join this exciting journey, shaping the future of GUI automation. In the spirit of open science, our code, data, and model will be made publicly available, paving the way for multimodal dialogue scenarios with intricate and precise interactions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
