arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.05147cs.CVcs.LGcs.ROstat.ML
收藏

实用的Grounded语言跨模态流形对齐

Practical Cross-modal Manifold Alignment for Grounded Language

University of Maryland, Baltimore County(马里兰大学巴尔的摩郡分校);Booz Allen Hamilton(博思艾伦汉密尔顿公司)

Andre T. Nguyen, Luke E. Richards, Gaoussou Youssouf Kebe, Edward Raff, Kasra Darvish, Frank Ferraro, Cynthia Matuszek

摘要:

我们提出了一个跨模态流形对齐程序,利用三重损失,共同学习一致的,多模态嵌入的语言为基础的概念的现实世界的项目。我们的方法通过从RGB深度图像及其自然语言描述中采样锚点、正数据点和负数据点的三元组来学习这些嵌入。我们表明,我们的方法可以受益于,但不需要,后处理步骤,如Procrustes分析,在对比我们的一些基线,需要它的合理性能。我们在两个通常用于开发基于机器人的接地语言学习系统的数据集上展示了我们方法的有效性,其中我们的方法在五个评估指标上优于四个基线,包括最先进的方法。

英文摘要:

We propose a cross-modality manifold alignment procedure that leverages triplet loss to jointly learn consistent, multi-modal embeddings of language-based concepts of real-world items. Our approach learns these embeddings by sampling triples of anchor, positive, and negative data points from RGB-depth images and their natural language descriptions. We show that our approach can benefit from, but does not require, post-processing steps such as Procrustes analysis, in contrast to some of our baselines which require it for reasonable performance. We demonstrate the effectiveness of our approach on two datasets commonly used to develop robotic-based grounded language learning systems, where our approach outperforms four baselines, including a state-of-the-art approach, across five evaluation metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑