While robot action planning based on large language models (LLMs) has advanced remarkably, continuing a task after a physical action failure during execution remains a key challenge. For example, after failing to grasp an object, the robot may not notice that it has fallen off the table and keep trying to detect and pick it up, stalling the task. Existing approaches either rely on closed-loop autonomous re-planning, which fails when errors exceed the robot's capabilities, or on human intervention without updating the LLM's plan, hindering subsequent planning. We present a system for LLM-based robot action planning that integrates remote human intervention with feedback-based replanning. When the LLM detects an action failure, the robot requests remote assistance; an operator resolves the failure through teleoperation and reports the outcome in natural language, which is fed back to the LLM to update its plan so that the task can continue. We evaluated the proposed system on a real-world trash-collection task, comparing it with a fully autonomous method and a teleoperation-only baseline. The proposed system improves task progress over the autonomous method in this case study. For easy-to-retrieve trash, the gap from the teleoperation reference was small, while a larger gap remained for harder-to-retrieve trash. These results suggest that, in this case study, combining remote human intervention with feedback-based replanning was associated with continued task execution even in the face of otherwise unrecoverable failures. The project website is https://emergentsystemlabstudent.github.io/REPAIR/.
Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging
Institute of Advanced Technology, University of Science and Technology of China(中国科学技术大学先进技术研究院);School of Information and Electronic, Beijing Institute of Technology(北京理工大学信息与电子学院);School of Mechanical Engineering, Tongji University(同济大学机械工程学院)
本文提出部署于Jetson Orin NX的视觉可穿戴转向辅助系统,通过轻量多任务网络检测跑道线与障碍物,并结合采样和样条曲线规划,帮助视障者安全慢跑。
Learning-Based Proxy Collision Detection for Robot Motion Planning Applications
Department of Electrical and Computer Engineering, University of California San Diego(加利福尼亚大学圣地亚哥分校电气与计算机工程系);University of California San Diego(加利福尼亚大学圣地亚哥分校);Electrical and Computer Engineering, University of California San Diego(加利福尼亚大学圣地亚哥分校电气与计算机工程)
Robust Spatiotemporal Motion Planning for Multi-Agent Autonomous Racing via Topological Gap Identification and Accelerated MPC
Department of Control Science and Engineering, Zhejiang University(浙江大学控制科学与工程学院);Center for Project-Based Learning, D-ITET, ETH Zurich(苏黎世联邦理工学院项目导向学习中心)
Ro-SLM: Onboard Small Language Models for Robot Task Planning and Operation Code Generation
Department of Computer & Information Science, University of Massachusetts Dartmouth(达特茅斯大学计算机与信息科学系);Department of Computer Science & Engineering, California State University San Marcos(加州州立大学桑马克斯分校计算机科学与工程系)
Follow-Bench: A Unified Motion Planning Benchmark for Socially-Aware Robot Person Following
Shenzhen Key Laboratory of Robotics and Computer Vision(深圳机器人与计算机视觉重点实验室);Southern University of Science and Technology(南方科技大学);Human-Robot Interfaces and Interaction Laboratory(人机交互实验室);Istituto Italiano Di Tecnologia(意大利技术研究院);Swiss Federal Technology Institute of Lausanne(洛桑联邦理工学院)
Melding LLM and temporal logic for reliable human-swarm collaboration in complex scenarios
School of Advanced Manufacturing and Robotics, Peking University(北京大学先进制造与机器人学院);Division of Natural and Applied Sciences, Duke Kunshan University(杜克昆山大学自然与应用科学学院);School of Automaton, Southeast University(东南大学自动化学院);School of Automation Science and Electrical Engineering, Beihang University(北航自动化科学与电气工程学院)
HygieneRoboBench: Benchmarking Hygiene-Aware Planning for Household Robots
Shanghai Jiao Tong University(上海交通大学);Eastern Institute of Technology, Ningbo(宁波东方理工大学);University of Waterloo(滑铁卢大学);Stony Brook University(纽约州立大学石溪分校);University of Science and Technology of China(中国科学技术大学);Chengdu Institute of Computer Applications, Chinese Academy of Sciences(中国科学院成都计算机应用研究所);Ningbo Institute of Digital Twin(宁波数字孪生研究院)