使用基于GAN的多模态目标源分类理解用于获取日常对象的自然语言指令
Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification
摘要：
在本文中，我们解决多模态语言理解的无约束取指令在国内服务机器人的背景。一个典型的提取指令，如“从白色的架子上给我拿黄色的玩具”，需要推断用户的意图，即提取什么对象（目标）和从哪里（源）。为了解决这个问题，我们提出了一个多模态目标-源分类器模型（MTCM），它预测场景中目标和源候选者的区域可能性。与其他方法不同，MTCM可以处理基于语言和视觉特征的区域分类。我们评估了我们的方法，该方法在标准数据集上优于最先进的方法。此外，我们使用生成对抗网络（MTCM-GAN）扩展了MTCM，并实现了同时的数据增强和分类。
英文摘要：
In this paper, we address multimodal language understanding for unconstrained fetching instruction in domestic service robots context. A typical fetching instruction such as "Bring me the yellow toy from the white shelf" requires to infer the user intention, that is what object (target) to fetch and from where (source). To solve the task, we propose a Multimodal Target-source Classifier Model (MTCM), which predicts the region-wise likelihood of target and source candidates in the scene. Unlike other methods, MTCM can handle regionwise classification based on linguistic and visual features. We evaluated our approach that outperformed the state-of-the-art method on a standard data set. In addition, we extended MTCM with Generative Adversarial Nets (MTCM-GAN), and enabled simultaneous data augmentation and classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
