arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.04218cs.ROcs.AIcs.CLcs.LG
收藏

小配对数据下描述与动作的双向翻译学习

Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data

Waseda University(早稻田大学);Fujitsu Limited(富士通有限公司);Waseda Research Institute for Science and Engineering(早稻田大学科学与工程研究院);National Institute of Advanced Industrial Science and Technology(日本产业技术综合研究所)

Minori Toyoda, Kanata Suzuki, Yoshihiko Hayashi, Tetsuya Ogata

摘要:

本研究使用来自不同模态的小配对数据实现了描述和动作之间的双向翻译。相互生成描述和动作的能力对于机器人在日常生活中与人类合作至关重要,这通常需要一个大型数据集来维护两种模态数据的全面配对。然而,一个配对的数据集是昂贵的建设和难以收集。为了解决这个问题,本研究提出了一个两阶段的双向翻译训练方法。在所提出的方法中,我们训练经常性的自编码器(RAE)的描述和行动与大量的非配对数据。然后,我们微调整个模型,使用小的配对数据绑定它们的中间表示。由于用于预训练的数据不需要配对,因此可以使用仅行为数据或大型语言语料库。我们使用由运动捕获动作和描述组成的配对数据集对我们的方法进行了实验评估。结果表明,我们的方法表现良好,即使当配对的数据量训练是小的。每个RAE的中间表示的可视化显示,类似的动作被编码在一个聚类的位置和相应的特征向量很好地对齐。

英文摘要:

This study achieved bidirectional translation between descriptions and actions using small paired data from different modalities. The ability to mutually generate descriptions and actions is essential for robots to collaborate with humans in their daily lives, which generally requires a large dataset that maintains comprehensive pairs of both modality data. However, a paired dataset is expensive to construct and difficult to collect. To address this issue, this study proposes a two-stage training method for bidirectional translation. In the proposed method, we train recurrent autoencoders (RAEs) for descriptions and actions with a large amount of non-paired data. Then, we finetune the entire model to bind their intermediate representations using small paired data. Because the data used for pre-training do not require pairing, behavior-only data or a large language corpus can be used. We experimentally evaluated our method using a paired dataset consisting of motion-captured actions and descriptions. The results showed that our method performed well, even when the amount of paired data to train was small. The visualization of the intermediate representations of each RAE showed that similar actions were encoded in a clustered position and the corresponding feature vectors were well aligned.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑