arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.09829cs.RO
收藏

VLMPC:面向机器人操作的视觉-语言模型预测控制

VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation

School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院)

Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, Wei Zhang

AI总结:

针对传统MPC缺乏环境感知能力在复杂机器人操作场景中失效的问题,提出融合VLM与MPC的VLMPC框架,通过条件动作采样、动作条件视频预测和分层代价函数生成最优动作,在公开基准和真实任务中表现更优。

摘要:

尽管模型预测控制(MPC)能有效预测系统的未来状态,因而被广泛应用于机器人操作任务,但它不具备环境感知能力,导致其在部分复杂场景中失效。为解决这一问题,我们提出视觉-语言模型预测控制(VLMPC),这是一个利用视觉语言模型(VLM)强大感知能力并将其与MPC融合的机器人操作框架。具体而言,我们提出了一个条件动作采样模块,该模块以目标图像或语言指令为输入,借助VLM采样得到一组候选动作序列。随后,我们设计了一个轻量级的动作条件视频预测模型,用于基于候选动作序列生成一组未来帧。VLMPC通过一个分层代价函数,在VLM的协助下生成最优动作序列,该代价函数同时刻画了当前观测与目标图像之间的像素级和知识级一致性。我们证明,VLMPC在公开基准上的表现优于现有最优方法。更重要的是,我们的方法在各类真实世界机器人操作任务中展现出优异性能。代码可在https://github.com/PPjmchen/VLMPC 获取。

英文摘要:

Although Model Predictive Control (MPC) can effectively predict the future states of a system and thus is widely used in robotic manipulation tasks, it does not have the capability of environmental perception, leading to the failure in some complex scenarios. To address this issue, we introduce Vision-Language Model Predictive Control (VLMPC), a robotic manipulation framework which takes advantage of the powerful perception capability of vision language model (VLM) and integrates it with MPC. Specifically, we propose a conditional action sampling module which takes as input a goal image or a language instruction and leverages VLM to sample a set of candidate action sequences. Then, a lightweight action-conditioned video prediction model is designed to generate a set of future frames conditioned on the candidate action sequences. VLMPC produces the optimal action sequence with the assistance of VLM through a hierarchical cost function that formulates both pixel-level and knowledge-level consistence between the current observation and the goal image. We demonstrate that VLMPC outperforms the state-of-the-art methods on public benchmarks. More importantly, our method showcases excellent performance in various real-world tasks of robotic manipulation. Code is available at~\url{https://github.com/PPjmchen/VLMPC}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑