arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.20876cs.RO
收藏

本体感知增强视觉语言模型在为机器人任务生成描述和子任务分割中的应用

Proprioception Enhances Vision Language Model in Generating Captions and Subtask Segmentations for Robot Task

Faculty of Science and Engineering, Waseda University(工学部,早稻田大学);Artificial Intelligence Laboratory, Fujitsu Limited(Fujitsu 人工智能实验室);National Institute of Advanced Industrial Science and Technology(国家先进工业科学与技术研究院)

Kanata Suzuki, Shota Shimizu, Tetsuya Ogata

AI总结:

本研究通过引入本体感知数据,提升视觉语言模型在机器人任务描述和子任务分割中的性能,以增强机器人模仿学习效率。

摘要:

从未来机器人发展的角度来看,验证仅在离线数据(如图像和语言)上训练的基础模型是否能理解机器人运动至关重要。特别是由于视觉语言模型(VLMs)在其训练数据集中不包含机器人低级运动信息,视频理解包括轨迹信息仍是一个重大挑战。在本研究中,我们通过一个包含低级机器人运动信息的视频描述任务评估了VLMs的两项能力:(1)自动描述机器人任务和(2)一系列任务的分割。这两种能力旨在通过将语言和运动联系起来提高机器人模仿学习的效率,并作为基础模型性能的衡量标准。所提出的方法通过图像描述和机器人任务轨迹数据生成多个“场景”描述。然后通过汇总这些单独描述生成完整的任务描述。此外,该方法通过比较图像描述文本嵌入的相似性来执行子任务分割。在两种描述任务中,所提出的方法旨在通过将机器人运动数据(关节和末端执行器状态)作为输入提供给VLM来提高性能。进行了模拟实验以验证所提出方法的有效性。

英文摘要:

From the perspective of future developments in robotics, it is crucial to verify whether foundation models trained exclusively on offline data, such as images and language, can understand the robot motion. In particular, since Vision Language Models (VLMs) do not include low-level motion information from robots in their training datasets, video understanding including trajectory information remains a significant challenge. In this study, we assess two capabilities of VLMs through a video captioning task with low-level robot motion information: (1) automatic captioning of robot tasks and (2) segmentation of a series of tasks. Both capabilities are expected to enhance the efficiency of robot imitation learning by linking language and motion and serve as a measure of the foundation model's performance. The proposed method generates multiple "scene" captions using image captions and trajectory data from robot tasks. The full task caption is then generated by summarizing these individual captions. Additionally, the method performs subtask segmentation by comparing the similarity between text embeddings of image captions. In both captioning tasks, the proposed method aims to improve performance by providing the robot's motion data - joint and end-effector states - as input to the VLM. Simulator experiments were conducted to validate the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑