arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.05674cs.RO
收藏

预训练视觉-语言模型在多种识别行为中的机器人应用

Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors

The University of Tokyo(东京大学)

Kento Kawaharazuka, Yoshiki Obinata, Naoaki Kanazawa, Kei Okada, Masayuki Inaba

AI总结:

本研究分类总结了灵活利用预训练视觉-语言模型的方法,使其在不直接连接机器人运动的情况下,支持运动选择与规划,并通过五种信息提取方法实现状态、物体、可供性、关系和异常识别,增强现有机器人的识别能力。

摘要:

近年来,许多从大型数据集中学习视觉与语言之间关系的模型已被发布。这些模型执行各种任务,例如回答关于图像的问题、检索与图像最匹配的句子,以及查找图像中与短语对应的区域。尽管有一些示例,这些预训练的视觉-语言模型与机器人之间的联系仍然薄弱。如果直接将它们连接到机器人运动,由于机器人的具身性和数据收集的困难,它们将失去通用性,并且无法适用于广泛的身体和情境。因此,在本研究中,我们分类并总结了如何灵活且容易地利用预训练的视觉-语言模型,以机器人能够理解的方式,而不直接将它们连接到机器人运动。我们讨论了如何在不重新训练模型的情况下,使用这些模型进行机器人运动选择和运动规划。我们考虑了五种提取机器人可理解信息的方法,并展示了基于这五种方法组合的状态识别、物体识别、可供性识别、关系识别和异常检测的结果。我们期望这项研究将为现有机器人的识别行为增加灵活性和易用性,以及新的应用。

英文摘要:

In recent years, a number of models that learn the relations between vision and language from large datasets have been released. These models perform a variety of tasks, such as answering questions about images, retrieving sentences that best correspond to images, and finding regions in images that correspond to phrases. Although there are some examples, the connection between these pre-trained vision-language models and robotics is still weak. If they are directly connected to robot motions, they lose their versatility due to the embodiment of the robot and the difficulty of data collection, and become inapplicable to a wide range of bodies and situations. Therefore, in this study, we categorize and summarize the methods to utilize the pre-trained vision-language models flexibly and easily in a way that the robot can understand, without directly connecting them to robot motions. We discuss how to use these models for robot motion selection and motion planning without re-training the models. We consider five types of methods to extract information understandable for robots, and show the results of state recognition, object recognition, affordance recognition, relation recognition, and anomaly detection based on the combination of these five methods. We expect that this study will add flexibility and ease-of-use, as well as new applications, to the recognition behavior of existing robots.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑