arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.15099cs.CL
收藏

MUG:用户界面上的交互式多模态Grounding

MUG: Interactive Multimodal Grounding on User Interfaces

Google Research(谷歌研究院)

Tao Li, Gang Li, Jingjie Zheng, Purple Wang, Yang Li

摘要:

我们提出了MUG,一种新的交互式任务,多模式接地,用户和代理在界面屏幕上协作工作。以前的作品在一个回合中建模了多模式UI基础:用户给出命令,代理响应命令。然而,在现实场景中,当目标动作固有地难以用自然语言表达时,用户命令可能是模糊的。MUG允许多轮交互,这样在看到代理响应时,用户可以为代理提供进一步的命令以改进甚至纠正其操作。这种交互对于提高现实世界用例中的基础性能至关重要。为了研究这个问题,我们创建了一个新的数据集,该数据集由77,820个移动界面上的人类用户-代理交互序列组成,其中20%涉及多轮交互。为了建立我们的基准,我们实验了一系列的建模变量和评估策略,包括离线和在线评估-在线策略包括人工评估和自动模拟器。我们的实验表明,允许迭代交互显着提高了18%的绝对任务完成在整个测试数据集和31%的挑战子集。我们的结果奠定了基础,进一步调查的问题。

英文摘要:

We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command. Yet, in a realistic scenario, a user command can be ambiguous when the target action is inherently difficult to articulate in natural language. MUG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to refine or even correct its actions. Such interaction is critical for improving grounding performances in real-world use cases. To investigate the problem, we create a new dataset that consists of 77,820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions. To establish our benchmark, we experiment with a range of modeling variants and evaluation strategies, including both offline and online evaluation-the online strategy consists of both human evaluation and automatic with simulators. Our experiments show that allowing iterative interaction significantly improves the absolute task completion by 18% over the entire test dataset and 31% over the challenging subset. Our results lay the foundation for further investigation of the problem.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑