arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.16405cs.RO
收藏

使用预训练视觉语言模型的视觉问答进行机器人二值状态识别

Binary State Recognition by Robots using Visual Question Answering of Pre-Trained Vision-Language Model

The University of Tokyo(东京大学)

Kento Kawaharazuka, Yoshiki Obinata, Naoaki Kanazawa, Kei Okada, Masayuki Inaba

AI总结:

本文提出将预训练视觉语言模型的视觉问答应用于机器人二值状态识别,无需重新训练即可用语言直观描述状态,并通过实验总结了提问与图像处理技术的特性。

摘要:

识别当前状态对于机器人的运行是不可或缺的。存在各种需要识别的状态,例如电梯门是开还是关,物体是否被正确抓取,以及电视是开还是关。迄今为止,这些状态通过程序化描述点云或原始图像的状态、对图像进行标注和学习、使用特殊传感器等方式来识别。与这些方法不同,我们将在大规模数据集上训练的预训练视觉语言模型(PTVLM)的视觉问答(VQA)应用于此类二值状态识别。这一想法使我们能够用语言直观地描述状态识别,而无需任何重新训练,从而以简单且通用的方式提升机器人的识别能力。我们总结了提问方法和图像处理的各种技术,并通过实验阐明了它们的特性。

英文摘要:

Recognition of the current state is indispensable for the operation of a robot. There are various states to be recognized, such as whether an elevator door is open or closed, whether an object has been grasped correctly, and whether the TV is turned on or off. Until now, these states have been recognized by programmatically describing the state of a point cloud or raw image, by annotating and learning images, by using special sensors, etc. In contrast to these methods, we apply Visual Question Answering (VQA) from a Pre-Trained Vision-Language Model (PTVLM) trained on a large-scale dataset, to such binary state recognition. This idea allows us to intuitively describe state recognition in language without any re-training, thereby improving the recognition ability of robots in a simple and general way. We summarize various techniques in questioning methods and image processing, and clarify their properties through experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑