For service mobile robots to be most effective, it must be possible for non-experts and even end-users to program them to do new tasks. Regardless of the programming method (e.g., by demonstration or traditional programming), robot task programs are challenging to write, because they rely on multiple actions to succeed, including human-robot interactions. Unfortunately, interactions are prone to fail, because a human may perform the wrong action (e.g., if the robot's request is not clear). Moreover, when the robot cannot directly observe the human action, it may not detect the failure until several steps after it occurs. Therefore, writing fault-tolerant robot tasks is beyond the ability of non-experts.
This paper presents a principled approach to detect and recover from a broad class of failures that occur in end-user programs on service mobile robots. We present a two-tiered Robot Task Programming Language (RTPL): 1) an expert roboticist uses a specification language to write a probabilistic model of the robot's actions and interactions, and 2) a non-expert then writes an ordinary sequential program for a particular task. The RTPL runtime system executes the task program sequentially, while using the probabilistic model to build a Bayesian network that tracks possible, unobserved failures. If an error is observed, RTPL uses Bayesian inference to find the likely root cause of the error, and then attempts to re-execute a portion of the program for recovery.
Our empirical results show that RTPL 1) allows complex tasks to be written concisely, 2) correctly identifies the root cause of failure, and 3) allows multiple tasks to recover from a variety of errors, without task-specific error-recovery code.
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
School of Information and Communication Engineering(信息与通信工程学院);Dalian University of Technology(大连理工大学);Beta Infinity(贝塔无限);Nanyang Technological University(南洋理工大学);Beijing Jiaotong University(北京交通大学)
Causal-History Test-Time Scaling for Failure Recovery in Autoregressive World-Action Models
The Hong Kong University of Science and Technology (HKUST)(香港科技大学);ACCESS – AI Chip Center for Emerging Smart Systems(ACCESS——新兴智能系统AI芯片中心);Zhejiang University(浙江大学)
FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
Nanyang Technological University(南洋理工大学);Centre for Frontier AI Research, A*STAR(A*STAR前沿人工智能研究中心);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)
Phase-Conditioned Imitation Learning with Autonomous Failure Recovery for Robust Deformable Object Manipulation
Department of Robotics, Tohoku University(东大理学院机器人系);JC STEM Lab of Robotics for Soft Materials, the Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Hong Kong(香港大学工程学院电气与计算机工程系软材料机器人实验室)
Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Alibaba DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Tsinghua University(清华大学);University of Technology Sydney(悉尼科技大学);University of Liverpool(利物浦大学)
Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis
Texas A&M University(德克萨斯A&M大学);University of Minnesota(明尼苏达大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Abaka AI(Abaka人工智能);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation
The Chinese University of Hong Kong(香港中文大学);Centre for Perceptual and Interactive Intelligence(感知与交互智能中心);Duke University(杜克大学);Sun Yat-sen University(中山大学);TengenX(天工科技);Tencent Robotics X(腾讯Robotics X实验室);Shenzhen Loop Area Institute(深圳河套学院);Guangdong Key Laboratory of Big Data Analysis and Processing(广东省大数据分析与处理重点实验室)
Self-Evolutionary Replanning for Failure-Aware Motion Planning
Department of Computer and Information Science, University of Macau(澳门大学计算机与信息科学系);SIAT, Chinese Academy of Sciences(中国科学院上海技术物理研究所);Department of Computer Science, University of Hong Kong(香港大学计算机科学系);Academy for Engineering & Technology, Fudan University(复旦大学工程与技术学院);Department of EEE, Southern University of Science and Technology(南方科技大学电子工程系)
Introspective Visuomotor Control: Exploiting Uncertainty in Deep Visuomotor Control for Failure Recovery
Applied AI Lab (A2I)(应用人工智能实验室(A2I));Dynamic Robot Systems (DRS)(动态机器人系统实验室(DRS));Oxford Robotics Institute (ORI), University of Oxford(牛津大学牛津机器人研究所(ORI))