使用视觉-语言 Transformer 的主动式人机交互
Proactive Human-Robot Interaction using Visuo-Lingual Transformers
AI总结:
本文提出 ViLing-MMT 视觉-语言多模态 Transformer,融合场景视觉、语言指令与物体交互知识,主动推断用户目标并建议中间任务。
摘要:
人类天生具备提取潜在视觉-语言线索以通过人际交互推断情境的能力。在协作过程中,这使得人们能够主动预测一系列任务背后的潜在意图。相比之下,与人类协作的机器人智能体只会机械地遵循基本指令来完成任务,或在为实现某一目标而工作时使用特定的人工设计触发条件来发起主动协作。赋予此类机器人推理最终目标并主动建议中间任务的能力,将产生一种更加直观的人机协作方法。为此,我们提出一种基于学习的方法,该方法利用来自场景的视觉线索、来自用户的语言指令以及先前物体-物体交互的知识,来识别并主动预测用户意图实现的潜在目标。具体而言,我们提出 ViLing-MMT,这是一种基于视觉-语言多模态 Transformer 的架构,可捕获模态间与模态内依赖关系,以提供准确的场景描述,并在适用时主动建议任务。我们在仿真和真实世界场景中评估所提出的模型。
英文摘要:
Humans possess the innate ability to extract latent visuo-lingual cues to infer context through human interaction. During collaboration, this enables proactive prediction of the underlying intention of a series of tasks. In contrast, robotic agents collaborating with humans naively follow elementary instructions to complete tasks or use specific hand-crafted triggers to initiate proactive collaboration when working towards the completion of a goal. Endowing such robots with the ability to reason about the end goal and proactively suggest intermediate tasks will engender a much more intuitive method for human-robot collaboration. To this end, we propose a learning-based method that uses visual cues from the scene, lingual commands from a user and knowledge of prior object-object interaction to identify and proactively predict the underlying goal the user intends to achieve. Specifically, we propose ViLing-MMT, a vision-language multimodal transformer-based architecture that captures inter and intra-modal dependencies to provide accurate scene descriptions and proactively suggest tasks where applicable. We evaluate our proposed model in simulation and real-world scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。