Large language models (LLMs) have gained increasing popularity in robotic task planning due to their exceptional abilities in text analytics and generation, as well as their broad knowledge of the world. However, they fall short in decoding visual cues. LLMs have limited direct perception of the world, which leads to a deficient grasp of the current state of the world. By contrast, the emergence of visual language models (VLMs) fills this gap by integrating visual perception modules, which can enhance the autonomy of robotic task planning. Despite these advancements, VLMs still face challenges, such as the potential for task execution errors, even when provided with accurate instructions. To address such issues, this paper proposes a ReplanVLM framework for robotic task planning. In this study, we focus on error correction interventions. An internal error correction mechanism and an external error correction mechanism are presented to correct errors under corresponding phases. A replan strategy is developed to replan tasks or correct error codes when task execution fails. Experimental results on real robots and in simulation environments have demonstrated the superiority of the proposed framework, with higher success rates and robust error correction capabilities in open-world tasks. Videos of our experiments are available at https://youtu.be/NPk2pWKazJc.
RA-DP: Rapid Adaptive Diffusion Policy for Training-Free High-frequency Robotics Replanning
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Noah’s Ark Laboratory, Huawei Technologies Canada(华为加拿大诺亚实验室);Department of Electrical and Computer Engineering, University of Alberta(阿尔伯塔大学电气与计算机工程系);Noah’s Ark Laboratory, Huawei Technologies(华为诺亚实验室)
OceanPlan: Hierarchical Planning and Replanning for Natural Language AUV Piloting in Large-scale Unexplored Ocean Environments
School of Electrical and Computer Engineering, Georgia Institute of Technology(佐治亚理工学院电气与计算机工程学院);The Hong Kong University of Science and Technology(香港科技大学);School of Electrical Engineering, University of Notre Dame(圣母大学电气工程学院)
Hong Kong University of Science and Technology(香港科技大学);Massachusetts Institute of Technology(麻省理工学院);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);Carnegie Mellon University(卡内基梅隆大学);UMass Amherst(马萨诸塞大学阿默斯特分校)
Online Information-Aware Motion Planning with Inertial Parameter Learning for Robotic Free-Flyers
SGT Inc.(SGT公司);NASA Ames Research Center(美国国家航空航天局艾姆斯研究中心);Institute for Systems and Robotics(系统与机器人研究所);Instituto Superior Técnico(里斯本高等理工学院);Massachusetts Institute of Technology(麻省理工学院)
A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoning
Human-Robot Interfaces and Interaction (HRII) Laboratory, Istituto Italiano di Tecnologia(人机交互(HRII)实验室,意大利技术研究院);Dept. of Informatics, Bioengineering, Robotics, and Systems Engineering, University of Genoa(信息学、生物工程、机器人学和系统工程系,热那亚大学)
Video Replanning via Latent Embedding Refinement and Rejection-Based Sampling
National Taiwan University(国立台湾大学);MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);Cornell University(康奈尔大学);Harvard University(哈佛大学);Department of XXX, University of YYY, Location, Country(XXX系,YYY大学,地点,国家);School of ZZZ, Institute of WWW, Location, Country(ZZZ学院,WWW研究所,地点,国家)
APTree: Extended Behavior Trees for Adaptive and Scalable Automated Robotic Task Planning in Construction
University of Stuttgart(斯图加特大学);Institute for Computational Design and Construction(计算设计与建造研究所);Institute for Control Engineering of Machine Tools and Manufacturing Units(机床与制造单元控制工程研究所)
Video-to-BT: Generating Reactive Behavior Trees from Human Demonstration Videos for Robotic Assembly
Munich Institute of Robotics and Machine Intelligence (MIRMI)(慕尼黑机器人与机器智能研究所);Technical University of Munich(慕尼黑技术大学);Aalto University(艾尔沃斯大学);Shanghai University(上海大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Robotic Test Tube Rearrangement Using Combined Reinforcement Learning and Motion Planning
Graduate School of Engineering Science, Osaka University(大阪大学工学研究科);H.U. Group Research Institute G.K.(H.U.集团综合研究所);National Inst. of AIST(日本产业技术综合研究所)
MPC-based Coarse-to-Fine Motion Planning for Robotic Object Transportation in Cluttered Environments
Department of Electrical and Computer Engineering, University of Kaiserslautern Landau(电气与计算机工程系,凯撒斯劳滕大学);Department of Mechanical Engineering, Dalhousie University(机械工程系,达尔豪斯大学)
AINav: Large Language Model-Based Adaptive Interactive Navigation
School of Advanced Manufacturing and Robotics, Peking University(北京大学先进制造与机器人学院);AI Institute, School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院人工智能研究所)
Warm-Starting Optimization-Based Motion Planning for Robotic Manipulators via Point Cloud-Conditioned Flow Matching
J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University(J. Mike Walker ’66 机械工程系,德克萨斯A&M大学);Zachry Department of Civil and Environmental Engineering, Texas A&M University(Zachry 市政与环境工程系,德克萨斯A&M大学)
AdaReP:Adaptive Re-Planning under Model Mismatch for Neural World-Model Predictive Control
Shanghai Jiao Tong University(上海交通大学);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院);University of Science and Technology of China(中国科学技术大学)
Adaptive Action Chunking at Inference-time for Vision-Language-Action Models
National University of Singapore(新加坡国立大学);Shenzhen University of Advanced Technology(深圳理工大学);Sangfor Technologies Inc.(深信服科技股份有限公司);Mininglamp Technology(明略科技);Shenzhen Technology University(深圳技术大学);City University of Hong Kong(香港城市大学)