UI-JEPA：通过屏幕上的用户活动实现对用户意图的主动感知
UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity
AI总结：
本文提出 UI-JEPA 自监督框架及 IIW、IIT 两个数据集，以 JEPA 式 UI 表示结合 LLM 解码器预测用户意图，在更低算力与延迟下达到并超过主流大型 MLLM。
摘要：
从一系列用户界面（UI）动作中生成用户意图，是全面 UI 理解中的一项核心挑战。多模态大语言模型（MLLM）的最新进展已在该领域带来显著进步，但它们对大量模型参数、计算能力和高延迟的需求，使其不适用于需要轻量级、低延迟或增强隐私的端侧解决方案场景。此外，高质量数据集的缺乏也阻碍了此类轻量级模型的发展。为应对这些挑战，我们提出 UI-JEPA，这是一种新颖框架，采用掩码策略，通过自监督学习从无标注数据中学习抽象 UI 嵌入，并结合为用户意图预测而微调的 LLM 解码器。我们还引入两个新的以 UI 为基础的多模态数据集：“Intent in the Wild”（IIW）和“Intent in the Tame”（IIT），分别面向少样本和零样本 UI 理解任务而设计。IIW 包含覆盖 219 个意图类别的 1.7K 个视频，而 IIT 包含覆盖 10 个类别的 914 个视频。我们为这些数据集建立了首批基线，结果表明，使用 JEPA 风格目标学习到的表示与 LLM 解码器相结合，能够取得与最先进大型 MLLM 相匹配的用户意图预测性能，同时显著减少标注和部署资源。以意图相似度分数衡量，在两个数据集的平均结果上，UI-JEPA 分别比 GPT-4 Turbo 和 Claude 3.5 Sonnet 高出 10.0% 和 7.2%。值得注意的是，UI-JEPA 在实现该性能的同时，在 IIW 数据集上将计算成本降低 50.5 倍，并将延迟改善 6.6 倍。这些结果强调了 UI-JEPA 的有效性，凸显了其在轻量级、高性能 UI 理解方面的潜力。
英文摘要：
Generating user intent from a sequence of user interface (UI) actions is a core challenge in comprehensive UI understanding. Recent advancements in multimodal large language models (MLLMs) have led to substantial progress in this area, but their demands for extensive model parameters, computing power, and high latency makes them impractical for scenarios requiring lightweight, on-device solutions with low latency or heightened privacy. Additionally, the lack of high-quality datasets has hindered the development of such lightweight models. To address these challenges, we propose UI-JEPA, a novel framework that employs masking strategies to learn abstract UI embeddings from unlabeled data through self-supervised learning, combined with an LLM decoder fine-tuned for user intent prediction. We also introduce two new UI-grounded multimodal datasets, "Intent in the Wild" (IIW) and "Intent in the Tame" (IIT), designed for few-shot and zero-shot UI understanding tasks. IIW consists of 1.7K videos across 219 intent categories, while IIT contains 914 videos across 10 categories. We establish the first baselines for these datasets, showing that representations learned using a JEPA-style objective, combined with an LLM decoder, can achieve user intent predictions that match the performance of state-of-the-art large MLLMs, but with significantly reduced annotation and deployment resources. Measured by intent similarity scores, UI-JEPA outperforms GPT-4 Turbo and Claude 3.5 Sonnet by 10.0% and 7.2% respectively, averaged across two datasets. Notably, UI-JEPA accomplishes the performance with a 50.5x reduction in computational cost and a 6.6x improvement in latency in the IIW dataset. These results underscore the effectiveness of UI-JEPA, highlighting its potential for lightweight, high-performance UI understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
