arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.10085cs.RO
收藏

基于CLIP特征、利用图像和文本实现面向多任务与多机器人的随机控制

CLIP feature-based randomized control using images and text for multiple tasks and robots

Toyota Central R&D Labs., Inc.(丰田中央研究所)

Kazuki Shibata, Hideki Deguchi, Shun Taguchi

AI总结:

本文提出将CLIP图像-文本相似度与随机控制结合的无策略学习框架,并通过仿真及两轮机器人、机械臂实验验证其跨任务、跨机器人有效性。

摘要:

本研究提出了一种利用视觉语言模型(VLM)面向多种任务和机器人的控制框架。值得注意的是,现有使用VLM的控制方法已在训练环境中的各类任务和机器人上取得高性能。然而,对于训练环境之外的任务和机器人,这些方法在学习控制策略时会产生高昂成本。考虑到工业和家用机器人的应用,在引入机器人的新环境中进行学习具有挑战性。为解决这一问题,我们提出了一种无需学习控制策略的控制框架。该框架将视觉语言CLIP模型与随机控制相结合。CLIP通过将图像和文本嵌入特征空间来计算二者之间的相似度。本研究采用CLIP计算相机图像与表示目标状态的文本之间的相似度。在我们的方法中,机器人由随机控制器控制,该控制器同时进行探索并提升相似度梯度。此外,我们对CLIP进行微调以提升所提方法的性能。最终,我们通过多任务仿真以及使用两轮机器人和机械臂开展的真实机器人实验,证实了本方法的有效性。

英文摘要:

This study presents a control framework leveraging vision language models (VLMs) for multiple tasks and robots. Notably, existing control methods using VLMs have achieved high performance in various tasks and robots in the training environment. However, these methods incur high costs for learning control policies for tasks and robots other than those in the training environment. Considering the application of industrial and household robots, learning in novel environments where robots are introduced is challenging. To address this issue, we propose a control framework that does not require learning control policies. Our framework combines the vision-language CLIP model with a randomized control. CLIP computes the similarity between images and texts by embedding them in the feature space. This study employs CLIP to compute the similarity between camera images and text representing the target state. In our method, the robot is controlled by a randomized controller that simultaneously explores and increases the similarity gradients. Moreover, we fine-tune the CLIP to improve the performance of the proposed method. Consequently, we confirm the effectiveness of our approach through a multitask simulation and a real robot experiment using a two-wheeled robot and robot arm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑