Complex object manipulation tasks often span over long sequences of operations. Task planning over long-time horizons is a challenging and open problem in robotics, and its complexity grows exponentially with an increasing number of subtasks. In this paper we present a deep learning network that learns dependencies and transitions across subtasks solely from a set of demonstration videos. We represent each subtask as an action symbol (e.g. move cup), and show that these symbols can be learned and predicted directly from image observations. Learning from demonstrations and visual observations are two main pillars of our approach. The former makes the learning tractable as it provides the network with information about the most frequent transitions and relevant dependency between subtasks (instead of exploring all possible combination), while the latter allows the network to continuously monitor the task progress and thus to interactively adapt to changes in the environment. We evaluate our framework on two long horizon tasks: (1) block stacking of puzzle pieces being executed by humans, and (2) a robot manipulation task involving pick and place of objects and sliding a cabinet door with a 7-DoF robot arm. We show that complex plans can be carried out when executing the robotic task and the robot can interactively adapt to changes in the environment and recover from failure cases.
Computer Science Department, RWTH Aachen University（鲁尔大学亚琛分校计算机科学系）；Cyber Physical Systems, Indian Institute of Science Bengaluru（印度科学研究院班加罗尔分校）；Alma Mater Studiorum Università di Bologna（博洛尼亚大学）；Cognitive Robotics Department, Delft University of Technology（代尔夫特理工大学认知机器人学系）
Long-horizon planning in realistic environments requires the ability to reason over sequential tasks in high-dimensional state spaces with complex dynamics. Classical motion planning algorithms, such as rapidly-exploring random trees, are capable of efficiently exploring large state spaces and computing long-horizon, sequential plans. However, these algorithms are generally challenged with complex, stochastic, and high-dimensional state spaces as well as in the presence of narrow passages, which naturally emerge in tasks that interact with the environment. Machine learning offers a promising s…
SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Model
Virginia Seafood Agricultural Research and Extension Center, and Department of Biological Systems Engineering, Virginia Tech（弗吉尼亚海产品农业研究中心及生物系统工程系，弗吉尼亚理工学院）；Department of Electrical and Computer Engineering, Drexel University（电气与计算机工程系，德雷塞尔大学）
Learning to enhance multi-legged robot on rugged landscapes
Institute for Robotics and Intelligent Machines, Georgia Institute of Technology（佐治亚理工学院机器人与智能机器研究所）；School of Physics, Georgia Institute of Technology（佐治亚理工学院物理学院）；School of Computing, Georgia Institute of Technology（佐治亚理工学院计算学院）
Polytechnique Montréal（蒙特利尔理工学院）；University of Oxford（牛津大学）；Desay SV Automotive（德赛西威汽车电子）
Motivated by the tremendous progress we witnessed in recent years, this paper presents a survey of the scientific literature on the topic of Collaborative Simultaneous Localization and Mapping (C-SLAM), also known as multi-robot SLAM. With fleets of self-driving cars on the horizon and the rise of multi-robot systems in industrial applications, we believe that Collaborative SLAM will soon become a cornerstone of future robotic applications. In this survey, we introduce the basic concepts of C-SLAM and present a thorough literature review. We also outline the major challenges and limitations of…
Integration of Robot and Scene Kinematics for Sequential Mobile Manipulation Planning
State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing（国家一般人工智能重点实验室，北京一般人工智能研究院（BIGAI），北京）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Institute for Artificial Intelligence and School of Artificial Intelligence, Peking University（北京大学人工智能研究院和人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Institute of Computer Engineering, TU Wien（计算机工程研究所，维也纳技术大学）；AIT Austrian Institute of Technology（奥地利技术研究所）；Institute of Mechanics and Mechatronics, TU Wien（机械与机电系统研究所，维也纳技术大学）
Task and Motion Planning has made great progress in solving hard sequential manipulation problems. However, a gap between such planning formulations and control methods for reactive execution remains. In this paper we propose a model predictive control approach dedicated to robustly execute a single sequence of constraints, which corresponds to a discrete decision sequence of a TAMP plan. We decompose the overall control problem into three sub-problems (solving for sequential waypoints, their timing, and a short receding horizon path) that each is a non-linear program solved online in each MPC…
Carnegie Mellon University（卡内基梅隆大学）；UC San Diego（加州大学圣地亚哥分校）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkil…
*The following abbreviates the abstract. Please refer to the thesis for the full abstract.* After a disaster, locating and extracting victims quickly is critical because mortality rises rapidly after the first two days. To assist search and rescue teams and improve response times, teams of camera-equipped aerial robots can engage in tasks such as mapping buildings and locating victims. These sensing tasks encapsulate difficult (NP-Hard) problems. One way to simplify planning for these tasks is to focus on maximizing sensing performance over a short time horizon. Specifically, consider the prob…
University of Michigan（密歇根大学）；Air Force Research Laboratory（空军研究实验室）
In this paper, we present a framework for trust-aware sequential decision-making in a human-robot team. We model the problem as a finite-horizon Markov Decision Process with a reward-based performance metric, allowing the robotic agent to make trust-aware recommendations. Results of a human-subject experiment show that the proposed trust update model is able to accurately capture the human agent's moment-to-moment trust changes. Moreover, we cluster the participants' trust dynamics into three categories, namely, Bayesian decision makers, oscillators, and disbelievers, and identify personal cha…
