arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.08253cs.RO
收藏

开放世界任务与运动规划 via 视觉-语言模型生成的约束

Open-World Task and Motion Planning via Vision-Language Model Generated Constraints

NVIDIA Research(NVIDIA研究);MIT CSAIL

Nishanth Kumar, William Shen, Fabio Ramos, Dieter Fox, Tomás Lozano-Pérez, Leslie Pack Kaelbling, Caelan Reed Garrett

AI总结:

本文提出OWL-TAMP方法,通过整合视觉-语言模型生成的约束,提升开放世界任务与运动规划的性能,实现对自然语言目标的处理能力。

摘要:

基于视觉-语言模型(VLMs)的基座模型在常见的视觉和语言任务上表现出色,例如视觉问答。然而,它们目前尚无法直接解决需要精确连续推理的复杂、长周期机器人操作问题。任务与运动规划(TAMP)系统通过在参数化技能上进行离散-连续混合搜索来处理长周期推理,但依赖于详细的环境模型,无法解释新的人类目标,如任意自然语言目标。我们提出通过让VLMs生成离散和连续的语言参数化约束来将VLMs整合到TAMP系统中,以实现开放世界推理。具体而言,我们利用VLMs生成离散的动作顺序约束,以约束TAMP在动作序列上的搜索,以及以代码形式存在的连续约束,以补充传统的TAMP操作约束。实验表明,我们的方法OWL-TAMP在多个直接用自然语言指定的长周期操作任务上优于仅依赖TAMP或VLMs的基线方法。我们还进一步证明,OWL-TAMP可以与现成的TAMP系统结合,解决在现实硬件上具有挑战性的操作任务。

英文摘要:

Foundation models like Vision-Language Models (VLMs) excel at common sense vision and language tasks such as visual question answering. However, they cannot yet directly solve complex, long-horizon robot manipulation problems requiring precise continuous reasoning. Task and Motion Planning (TAMP) systems can handle long-horizon reasoning through discrete-continuous hybrid search over parameterized skills, but rely on detailed environment models and cannot interpret novel human objectives, such as arbitrary natural language goals. We propose integrating VLMs into TAMP systems by having them generate discrete and continuous language-parameterized constraints that enable open-world reasoning. Specifically, we use VLMs to generate discrete action ordering constraints that constrain TAMP search over action sequences, and continuous constraints in the form of code that augments traditional TAMP manipulation constraints. Experiments show that our approach, OWL-TAMP, outperforms baselines relying solely on TAMP or VLMs across several long-horizon manipulation tasks specified directly in natural language. We additionally demonstrate that OWL-TAMP can be deployed with an off-the-shelf TAMP system to solve challenging manipulation tasks on real-world hardware.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑