UGIF：基于UI落地的指令遵循
UGIF: UI Grounded Instruction Following
AI总结：
本文提出UGIF-DataSet，一个多语言多模态UI落地数据集，用于智能手机分步任务完成，并基于LLMs提出跨模态跨语言检索与指令解析方法，发现英文UI任务完成率为48%，其他语言降至32%。
摘要：
智能手机用户常常发现，在海量的菜单中进行导航以执行诸如“如何拦截未知号码来电？”之类的常见任务是很困难的。目前，带有分步说明的帮助文档是手动编写的，以帮助用户。通过将帮助文档中的说明落地到UI，并在手机UI上叠加教程，可以进一步提升用户体验。为了构建此类教程，需要包括检索、解析和落地在内的多个自然语言处理组件，但目前尚无用于此类任务的任何相关数据集。因此，我们引入了UGIF-DataSet，这是一个多语言、多模态的基于UI落地的数据集，用于在智能手机上完成分步任务，包含跨越8种语言的4,184个任务。作为该问题的初步方法，我们提出根据用户的查询检索相关的指令步骤，并使用大型语言模型（LLMs）解析这些步骤，以生成可在设备上执行的宏。指令步骤通常仅提供英文版本，因此挑战包括从多种语言的用户查询中进行跨模态、跨语言的英文操作指南页面检索，以及将英文指令步骤映射到可能是不同语言的UI上。我们比较了包括PaLM和GPT-3在内的不同LLMs的性能，发现英文UI的端到端任务完成率为48%，但其他语言的性能下降至32%。我们分析了现有模型在此任务上的常见失败模式，并指出了改进方向。
英文摘要：
Smartphone users often find it difficult to navigate myriad menus to perform common tasks such as "How to block calls from unknown numbers?". Currently, help documents with step-by-step instructions are manually written to aid the user. The user experience can be further enhanced by grounding the instructions in the help document to the UI and overlaying a tutorial on the phone UI. To build such tutorials, several natural language processing components including retrieval, parsing, and grounding are necessary, but there isn't any relevant dataset for such a task. Thus, we introduce UGIF-DataSet, a multi-lingual, multi-modal UI grounded dataset for step-by-step task completion on the smartphone containing 4,184 tasks across 8 languages. As an initial approach to this problem, we propose retrieving the relevant instruction steps based on the user's query and parsing the steps using Large Language Models (LLMs) to generate macros that can be executed on-device. The instruction steps are often available only in English, so the challenge includes cross-modal, cross-lingual retrieval of English how-to pages from user queries in many languages and mapping English instruction steps to UI in a potentially different language. We compare the performance of different LLMs including PaLM and GPT-3 and find that the end-to-end task completion rate is 48% for English UI but the performance drops to 32% for other languages. We analyze the common failure modes of existing models on this task and point out areas for improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
