Complex manipulation tasks can contain various execution branches of primitive skills in sequence or in parallel under different scenarios. Manual specifications of such branching conditions and associated skill parameters are not only error-prone due to corner cases but also quickly untraceable given a large number of objects and skills. On the other hand, learning from demonstration has increasingly shown to be an intuitive and effective way to program such skills for industrial robots. Parameterized skill representations allow generalization over new scenarios, which however makes the planning process much slower thus unsuitable for online applications. In this work, we propose a hierarchical and compositional planning framework that learns a Geometric Task Network (GTN) from exhaustive planners, without any manual inputs. A GTN is a goal-dependent task graph that encapsulates both the transition relations among skill representations and the geometric constraints underlying these transitions. This framework has shown to improve dramatically the offline learning efficiency, the online performance and the transparency of decision process, by leveraging the task-parameterized models. We demonstrate the approach on a 7-DoF robot arm both in simulation and on hardware solving various manipulation tasks.
College of Engineering, Peking University（北京大学工学院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）
Learning from demonstration (LfD) provides a fast, intuitive and efficient framework to program robot skills, which has gained growing interest both in research and industrial applications. Most complex manipulation tasks are long-term and involve a set of skill primitives. Thus it is crucial to have a reliable coordination scheme that selects the correct sequence of skill primitive and the correct parameters for each skill, under various scenarios. Instead of relying on a precise simulator, this work proposes a human-in-the-loop coordination framework for LfD skills that: builds parameterized…
Imperial College London（伦敦帝国理工学院）；Tencent Robotics X（腾讯 Robotics X）；Bielefeld University（比勒费尔德大学）
To accurately pour drinks into various containers is an essential skill for service robots. However, drink pouring is a dynamic process and difficult to model. Traditional deep imitation learning techniques for implementing autonomous robotic pouring have an inherent black-box effect and require a large amount of demonstration data for model training. To address these issues, an Explainable Hierarchical Imitation Learning (EHIL) method is proposed in this paper such that a robot can learn high-level general knowledge and execute low-level actions across multiple drink pouring scenarios. Moreov…
Imitation Learning (IL) is an effective framework to learn visuomotor skills from offline demonstration data. However, IL methods often fail to generalize to new scene configurations not covered by training data. On the other hand, humans can manipulate objects in varying conditions. Key to such capability is hand-eye coordination, a cognitive ability that enables humans to adaptively direct their movements at task-relevant objects and be invariant to the objects' absolute spatial location. In this work, we present a learnable action space, Hand-eye Action Networks (HAN), that can approximate …
This paper presents JEDAI, an AI system designed for outreach and educational efforts aimed at non-AI experts. JEDAI features a novel synthesis of research ideas from integrated task and motion planning and explainable AI. JEDAI helps users create high-level, intuitive plans while ensuring that they will be executable by the robot. It also provides users customized explanations about errors and helps improve their understanding of AI planning as well as the limits and capabilities of the underlying robot system.
We study the problem of learning manipulation skills from human demonstration video by inferring the association relationships between geometric features. Motivation for this work stems from the observation that humans perform eye-hand coordination tasks by using geometric primitives to define a task while a geometric control error drives the task through execution. We propose a graph based kernel regression method to directly infer the underlying association constraints from human demonstration video using Incremental Maximum Entropy Inverse Reinforcement Learning (InMaxEnt IRL). The learned …
International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术研究所）
With the advent of artificial intelligence and machine learning, humanoid robots are made to learn a variety of skills which humans possess. One of fundamental skills which humans use in day-to-day activities is performing tasks with coordination between both the hands. In case of humanoids, learning such skills require optimal motion planning which includes avoiding collisions with the surroundings. In this paper, we propose a framework to learn coordinated tasks in cluttered environments based on DiGrad - A multi-task reinforcement learning algorithm for continuous action-spaces. Further, we…
Microsoft（微软）；Tokyo Institute of Technology（东京工业大学）
A challenge in robot grasping is to achieve task-grasping which is to select a grasp that is advantageous to the success of tasks before and after grasps. One of the frameworks to address this difficulty is Learning-from-Observation (LfO), which obtains various hints from human demonstrations. This paper solves three issues in the grasping skills in the LfO framework: 1) how to functionally mimic human-demonstrated grasps to robots with limited grasp capability, 2) how to coordinate grasp skills with reaching body mimicking, 3) how to robustly perform grasps under object pose and shape uncerta…
Ag2x2: Robust Agent-Agnostic Visual Representations for Zero-Shot Bimanual Manipulation
National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（国家关键人工智能实验室，北京通用人工智能研究院（BIGAI））；School of Psychological and Cognitive Sciences, Peking University（北京大学心理与认知科学学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Beijing Key Laboratory of Behavior and Mental Health, Peking University（北京行为与心理健康重点实验室，北京大学）；Yuanpei College, Peking University（北京大学元培学院）；Department of Computer Science and Technology, University of Cambridge（剑桥大学计算机科学与技术系）；Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence（具身智能实验室，北京大学-武汉人工智能研究院）
EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation
OpenDriveLab at The University of Hong Kong（香港大学OpenDriveLab）；Alibaba Group（阿里巴巴集团）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；Beihang University（北京航空航天大学）；Sichuan University（四川大学）；Archon Robotics（Archon机器人公司）
AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；RoboScience（机器人科学）；East China Normal University（华东师范大学）；Peking University（北京大学）；Nanjing University（南京大学）
ViLoMan: Learning Visual-Proprioceptive Whole-Body Loco-Manipulation Skills for Humanoid Robots
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences (CAS)（中国科学院大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京智源人工智能研究院）
Human-AI Interaction in Industrial Robotics: Design and Empirical Evaluation of a User Interface for Explainable AI-Based Robot Program Optimization
ArtiMinds Robotics（阿蒂迈因兹机器人公司）；Institute for Artificial Intelligence, University of Bremen（不来梅大学人工智能研究所）；wbk Institute of Production Science, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院生产科学wbk研究所）；Institute for Learning and Innovation in Networks, Karlsruhe University of Applied Sciences（卡尔斯鲁厄应用科学大学网络学习与创新研究所）
Human and robot partners increasingly need to work together to perform tasks as a team. Robots designed for such collaboration must reason about how their task-completion strategies interplay with the behavior and skills of their human team members as they coordinate on achieving joint goals. Our goal in this work is to develop a computational framework for robot adaptation to human partners in human-robot team collaborations. We first present an algorithm for autonomously recognizing available task-completion strategies by observing human-human teams performing a collaborative task. By transf…
Learning to Recover: Dynamic Reward Shaping with Wheel-Leg Coordination for Fallen Robots
Humanoids and Human-Centered Mechatronics (HHCM), Istituto Italiano di Tecnologia（人类oids与人本机电系统（HHCM），意大利技术研究所）；Ph.D. Program of National Interest in Robotics and Intelligent Machines (DRIM), University of Genova（机器人与智能机器国家利益博士项目（DRIM），热那亚大学）；Department of Computer Science, University College London（计算机科学系，伦敦大学学院）
The act of reaching for an object is a fundamental yet complex skill for a robotic agent, requiring a high degree of visuomotor control and coordination. In consideration of dynamic environments, a robot capable of autonomously adapting to novel situations is desired. In this paper, a developmental robotics approach is used to autonomously learn visuomotor coordination on the NICO (Neuro-Inspired COmpanion) platform, for the task of object reaching. The robot interacts with its environment and learns associations between motor commands and temporally correlated sensory perceptions based on Heb…
