强化UI指令 grounding:迈向通用UI任务自动化API
Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
AI总结:
针对现有领域模型/API难以覆盖多样UI交互自动化需求的问题,本研究构建多模态强化UI指令grounding模型,结合像素到序列范式与强化学习算法,实现通用UI任务自动化,性能显著优于SOTA方法。
摘要:
大语言模型(LLMs)近期的普及为连接LLM与各类领域特定模型或API以实现众多AI任务自动化开辟了无限可能,其中LLM充当调度器,领域特定模型或API则作为动作执行器。尽管领域特定模型/API数量庞大,但它们仍难以全面覆盖用户与用户界面(UI)交互中极其多样的自动化需求。在本研究中,我们构建了一个多模态模型,用于将自然语言指令在给定UI截图中进行grounding,以此作为通用UI任务自动化执行器。这个无元数据的grounding模型由视觉编码器和语言解码器组成,首先在已被充分研究的文档理解任务上进行预训练,随后学习以可提示的方式从UI截图中解码空间信息。为了促进图像到文本预训练知识的利用,我们遵循像素到序列范式,使用语言解码器以token序列的形式预测几何坐标。我们进一步提出了一种创新的基于强化学习(RL)的算法,结合视觉语义指标对该序列中的token进行联合监督,有效增强了像素到序列范式的空间解码能力。大量实验表明,我们提出的强化UI指令grounding模型显著优于现有最优方法,并展现出作为通用UI任务自动化API的潜力。
英文摘要:
Recent popularity of Large Language Models (LLMs) has opened countless possibilities in automating numerous AI tasks by connecting LLMs to various domain-specific models or APIs, where LLMs serve as dispatchers while domain-specific models or APIs are action executors. Despite the vast numbers of domain-specific models/APIs, they still struggle to comprehensively cover super diverse automation demands in the interaction between human and User Interfaces (UIs). In this work, we build a multimodal model to ground natural language instructions in given UI screenshots as a generic UI task automation executor. This metadata-free grounding model, consisting of a visual encoder and a language decoder, is first pretrained on well studied document understanding tasks and then learns to decode spatial information from UI screenshots in a promptable way. To facilitate the exploitation of image-to-text pretrained knowledge, we follow the pixel-to-sequence paradigm to predict geometric coordinates in a sequence of tokens using a language decoder. We further propose an innovative Reinforcement Learning (RL) based algorithm to supervise the tokens in such sequence jointly with visually semantic metrics, which effectively strengthens the spatial decoding capability of the pixel-to-sequence paradigm. Extensive experiments demonstrate our proposed reinforced UI instruction grounding model outperforms the state-of-the-art methods by a clear margin and shows the potential as a generic UI task automation API.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。