arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.08655cs.HCcs.AI
收藏

使用大语言模型实现与移动UI的对话交互

Enabling Conversational Interaction with Mobile UI using Large Language Models

University of Toronto(多伦多大学);Google Research(谷歌研究院)

Bryan Wang, Gang Li, Yang Li

摘要:

会话代理显示出允许用户使用语言与移动设备交互的前景。然而,为了使用自然语言执行各种UI任务,开发人员通常需要为每个特定任务创建单独的数据集和模型,这是昂贵且耗时的。最近,预训练的大语言模型(LLM)已被证明能够推广到各种下游任务时,提示与目标任务的一些例子。本文研究了使用单个LLM实现与移动UI的通用会话交互的可行性。我们设计了提示技术,使LLM适应移动UI。我们尝试了四个重要的建模任务,这些任务解决了会话交互中的各种场景。我们的方法在这些具有挑战性的任务上取得了有竞争力的性能,而不需要专门的数据集和训练,提供了一种轻量级和可推广的方法来实现基于语言的移动交互。

英文摘要:

Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We experimented with four important modeling tasks that address various scenarios in conversational interaction. Our method achieved competitive performance on these challenging tasks without requiring dedicated datasets and training, offering a lightweight and generalizable approach to enable language-based mobile interaction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑