AFRAgent:一种基于自适应特征归一化的高分辨率感知GUI代理
AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent
AI总结:
AFRAgent通过自适应特征归一化技术,在保持高分辨率细节的同时,实现了更高效的GUI自动化性能,比现有模型小四分之一。
摘要:
随着移动用户界面(UI)自动化需求的增加,其在各行业中的广泛应用推动了技术发展。随着视觉语言模型(VLMs)的出现,GUI自动化已从为人类生成基于文本的指令发展到能够自主执行任务,从而优化自动化流程。最近的方法利用VLMs来解决这个问题,因为它们能够1)直接处理屏幕上的内容,2)通过利用人类行为(如点击、输入)而不依赖设备特定的API,3)应用现实世界的情境知识来理解任务。然而,这些模型在准确识别小部件和确定操作方面存在困难,因为视觉编码器特征中的空间信息有限。此外,表现最佳的模型通常较大,需要大量的训练,导致推理延迟。在本工作中,我们引入了AFRAgent,一种基于instruct-BLIP的多模态架构,其在GUI自动化中的性能优于其最近的竞争对手,且大小仅为其四分之一。为了增强大语言模型(LLM)管道中的图像嵌入,我们提出了一种基于自适应特征归一化的(基于token级仿射变换)技术,该技术有效丰富了低分辨率图像嵌入并融合了高分辨率细节。我们评估了AFRAgent在Meta-GUI和AITW基准上的表现,建立了智能手机自动化的新的最先进基线。
英文摘要:
There is a growing demand for mobile user interface (UI) automation, driven by its broad applications across industries. With the advent of visual language models (VLMs), GUI automation has progressed from generating text-based instructions for humans to autonomously executing tasks, thus optimizing automation workflows. Recent approaches leverage VLMs for this problem due to their ability to 1) process on-screen content directly, 2) remain independent of device-specific APIs by utilizing human actions (e.g., clicks, typing), and 3) apply real-world contextual knowledge for task understanding. However, these models often have trouble accurately identifying widgets and determining actions due to limited spatial information in vision encoder features. Additionally, top-performing models are often large, requiring extensive training and resulting in inference delays. In this work, we introduce AFRAgent, an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation while being less than one-fourth the size of its nearest competitor. To enhance image embeddings in the large language model (LLM) pipeline, we propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings and fuses high-resolution details. We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。