Deformable objects present a formidable challenge for robotic manipulation due to the lack of canonical low-dimensional representations and the difficulty of capturing, predicting, and controlling such objects. We construct compact topological representations to capture the state of highly deformable objects that are topologically nontrivial. We develop an approach that tracks the evolution of this topological state through time. Under several mild assumptions, we prove that the topology of the scene and its evolution can be recovered from point clouds representing the scene. Our further contribution is a method to learn predictive models that take a sequence of past point cloud observations as input and predict a sequence of topological states, conditioned on target/future control actions. Our experiments with highly deformable objects in simulation show that the proposed multistep predictive models yield more precise results than those obtained from computational topology libraries. These models can leverage patterns inferred across various objects and offer fast multistep predictions suitable for real-time applications.
Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a pre-computed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and e…
We demonstrate model-based, visual robot manipulation of linear deformable objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is…
University of California, Berkeley（加州大学伯克利分校）；New York University（纽约大学）
Using visual model-based learning for deformable object manipulation is challenging due to difficulties in learning plannable visual representations along with complex dynamic models. In this work, we propose a new learning framework that jointly optimizes both the visual representation model and the dynamics model using contrastive estimation. Using simulation data collected by randomly perturbing deformable objects on a table, we learn latent dynamics models for these objects in an offline fashion. Then, using the learned models, we use simple model-based planning to solve challenging deform…
University of California, Berkeley（加州大学伯克利分校）；NVIDIA Corporation（英伟达公司）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
Robotic grasping of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, developing grasp strategies for such objects is uniquely challenging. In this work, we efficiently simulate grasps on a wide range of 3D deformable objects using a GPU-based implementation of the corotational finite element method (FEM). To facilitate future research, we open-source our simulated dataset (34 objects, 1e5 Pa elasticity range, 6800 grasp evaluations, 1.1M grasp meas…
Robotic manipulation of deformable objects remains a challenging task. One such task is folding a garment autonomously. Given start and end folding positions, what is an optimal trajectory to move the robotic arm to fold a garment? Certain trajectories will cause the garment to move, creating wrinkles, and gaps, other trajectories will fail altogether. We present a novel solution to find an optimal trajectory that avoids such problematic scenarios. The trajectory is optimized by minimizing a quadratic objective function in an off-line simulator, which includes material properties of the garmen…
Carnegie Mellon University（卡内基梅隆大学）；UC San Diego（加州大学圣地亚哥分校）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkil…
Reinforcement Learning Compensated Model Predictive Control for Off-road Driving on Unknown Deformable Terrain
Department of Automotive Engineering, Clemson University（汽车工程系，克莱姆森大学）；Ground Vehicle Systems Center, U.S. Army Combat Capabilities Development Command（地面车辆系统中心，美国陆军作战能力发展指挥部）
Actor-Critic Cooperative Compensation to Model Predictive Control for Off-Road Autonomous Vehicles Under Unknown Dynamics
Department of Automotive Engineering, Clemson University（汽车工程系，克莱姆森大学）；Ground Vehicle Systems Center, U.S. Army Combat Capabilities Development Command（地面车辆系统中心，美国陆军作战能力发展指挥部）
Nonlinear Predictive Control of the Continuum and Hybrid Dynamics of a Suspended Deformable Cable for Aerial Pick and Place
Department of Computer, Control and Management Engineering, Sapienza University of Rome（计算机、控制与管理工程系，罗马萨皮恩扎大学）；Robotics and Mechatronics Department, Electrical Engineering, Mathematics, and Computer Science (EEMCS) Faculty, University of Twente（机器人与机电系，电气工程、数学与计算机科学（EEMCS） faculty，特文特大学）
Manipulating deformable objects, such as ropes and clothing, is a long-standing challenge in robotics, because of their large degrees of freedom, complex non-linear dynamics, and self-occlusion in visual perception. The key difficulty is a suitable representation, rich enough to capture the object shape, dynamics for manipulation and yet simple enough to be estimated reliably from visual observations. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). G-DOOM approximates a deformable object as a sparse set of interacting keypoints, which are extracted au…
Kuaishou Technology（快手科技）；Cornell University（康奈尔大学）；NVIDIA（英伟达公司）；Tencent（腾讯）；University of California, Irvine（加利福尼亚大学欧文分校）
Registration is a fundamental task in medical robotics and is often a crucial step for many downstream tasks such as motion analysis, intra-operative tracking and image segmentation. Popular registration methods such as ANTs and NiftyReg optimize objective functions for each pair of images from scratch, which are time-consuming for 3D and sequential images with complex deformations. Recently, deep learning-based registration approaches such as VoxelMorph have been emerging and achieve competitive performance. In this work, we construct a test-time training for deep deformable image registratio…
Certifiably Safe Manipulation of Deformable Linear Objects via Joint Shape and Tension Prediction
Department of Robotics, University of Michigan（机器人学系，密歇根大学）；Department of Mechanical Science and Engineering, University of Illinois at Urbana-Champaign（机械科学与工程系，伊利诺伊大学厄巴纳-香槟分校）
Planning-Query-Guided Model Generation for Model-Based Deformable Object Manipulation
Department of Computer Science, Stevens Institute of Technology（斯蒂文斯理工学院计算机科学系）；Department of Robotics, University of Michigan（密歇根大学机器人学系）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Towards Safe Learning-Based Non-Linear Model Predictive Control through Recurrent Neural Network Modeling
Institute of Computer Engineering, TU Wien（计算机工程研究所，维也纳技术大学）；AIT Austrian Institute of Technology（奥地利技术研究所）；Institute of Mechanics and Mechatronics, TU Wien（机械与机电系统研究所，维也纳技术大学）
Robot Deformable Object Manipulation via NMPC-generated Demonstrations in Deep Reinforcement Learning
Hubei Key Laboratory of Brain-inspired Intelligent Systems and the Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（湖北脑启发智能系统重点实验室和图像处理与智能控制重点实验室，人工智能与自动化学院，华中科技大学）
Hierarchical Task Model Predictive Control for Sequential Mobile Manipulation Tasks
Learning Systems and Robotics Lab（学习系统与机器人实验室）；Technical University of Munich（慕尼黑技术大学）；University of Toronto Institute for Aerospace Studies（多伦多大学航空航天研究所）；University of Toronto Robotics Institute（多伦多大学机器人研究所）；Munich Institute of Robotics and Machine Intelligence（慕尼黑机器人与机器智能研究所）；Vector Institute for Artificial Intelligence（人工智能向量研究所）
Learning Multimodal Attention for Manipulating Deformable Objects with Changing States
Future Robotics Organization, Waseda University（早稻田大学未来机器人组织）；Microsoft Research Asia（微软亚洲研究院）；Department of Modern Mechanical Engineering, Waseda University（早稻田大学现代机械工程系）；Artificial Intelligence Laboratories, Fujitsu Limited（Fujitsu 人工智能实验室）；Center for Technology Innovation - Controls and Robotics, Research & Development Group, Hitachi, Ltd.（富士通技术研发集团技术创新中心 - 控制与机器人）；Faculty of Science and Engineering, Waseda University（早稻田大学工学部）；National Institute of Advanced Science and Technology（国家先进科学研究院）
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；Universitat Politècnica de Catalunya - BarcelonaTECH（加泰罗尼亚理工大学-巴塞罗那理工学院）
Robotic cloth manipulation is a relevant challenging problem for autonomous robotic systems. Highly deformable objects as textile items can adopt multiple configurations and shapes during their manipulation. Hence, robots should not only understand the current cloth configuration but also be able to predict the future possible behaviors of the cloth. This paper addresses the problem of indirectly controlling the configuration of certain points of a textile object, by applying actions on other parts of the object through the use of a Model Predictive Control (MPC) strategy, which also allows to…
