We present a framework for deformable object manipulation that interleaves planning and control, enabling complex manipulation tasks without relying on high-fidelity modeling or simulation. The key question we address is when should we use planning and when should we use control to achieve the task? Planners are designed to find paths through complex configuration spaces, but for highly underactuated systems, such as deformable objects, achieving a specific configuration is very difficult even with high-fidelity models. Conversely, controllers can be designed to achieve specific configurations, but they can be trapped in undesirable local minima due to obstacles. Our approach consists of three components: (1) A global motion planner to generate gross motion of the deformable object; (2) A local controller for refinement of the configuration of the deformable object; and (3) A novel deadlock prediction algorithm to determine when to use planning versus control. By separating planning from control we are able to use different representations of the deformable object, reducing overall complexity and enabling efficient computation of motion. We provide a detailed proof of probabilistic completeness for our planner, which is valid despite the fact that our system is underactuated and we do not have a steering function. We then demonstrate that our framework is able to successfully perform several manipulation tasks with rope and cloth in simulation which cannot be performed using either our controller or planner alone. These experiments suggest that our planner can generate paths efficiently, taking under a second on average to find a feasible path in three out of four scenarios. We also show that our framework is effective on a 16 DoF physical robot, where reachability and dual-arm constraints make the planning more difficult.
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；City University of Hong Kong（香港城市大学）
The complex physical properties of highly deformable materials such as clothes pose significant challenges fanipulation systems. We present a novel visual feedback dictionary-based method for manipulating defoor autonomous robotic mrmable objects towards a desired configuration. Our approach is based on visual servoing and we use an efficient technique to extract key features from the RGB sensor stream in the form of a histogram of deformable model features. These histogram features serve as high-level representations of the state of the deformable material. Next, we collect manipulation data …
Real-life control tasks involve matters of various substances---rigid or soft bodies, liquid, gas---each with distinct physical behaviors. This poses challenges to traditional rigid-body physics engines. Particle-based simulators have been developed to model the dynamics of these complex scenes; however, relying on approximation techniques, their simulation often deviates from real-world physics, especially in the long term. In this paper, we propose to learn a particle-based simulator for complex control tasks. Combining learning with particle-based systems brings in two major benefits: first…
Manipulating deformable objects has long been a challenge in robotics due to its high dimensional state representation and complex dynamics. Recent success in deep reinforcement learning provides a promising direction for learning to manipulate deformable objects with data driven methods. However, existing reinforcement learning benchmarks only cover tasks with direct state observability and simple low-dimensional dynamics or with relatively simple image-based environments, such as those with rigid objects. In this paper, we present SoftGym, a set of open-source simulated benchmarks for manipu…
Stanford University（斯坦福大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；NVIDIA（英伟达）；Caltech（加州理工学院）
Manipulating volumetric deformable objects in the real world, like plush toys and pizza dough, bring substantial challenges due to infinite shape variations, non-rigid motions, and partial observability. We introduce ACID, an action-conditional visual dynamics model for volumetric deformable objects based on structured implicit neural representations. ACID integrates two new techniques: implicit representations for action-conditional dynamics and geodesics-based contrastive learning. To represent deformable dynamics from partial RGB-D observations, we learn implicit representations of occupanc…
We demonstrate model-based, visual robot manipulation of linear deformable objects. Our approach is based on a state-space representation of the physical system that the robot aims to control. This choice has multiple advantages, including the ease of incorporating physics priors in the dynamics model and perception model, and the ease of planning manipulation actions. In addition, physical states can naturally represent object instances of different appearances. Therefore, dynamics in the state space can be learned in one setting and directly used in other visually different settings. This is…
Regulating contact forces with high precision is crucial for grasping and manipulating fragile or deformable objects. We aim to utilize the dexterity of human hands to regulate the contact forces for robotic hands and exploit human sensory-motor synergies in a wearable and non-invasive way. We extracted force information from the electric activities of skeletal muscles during their voluntary contractions through surface electromyography (sEMG). We built a regression model based on a Neural Network to predict the gripping force from the preprocessed sEMG signals and achieved high accuracy (R2 =…
Dexterous Manipulation of Deformable Objects via Pneumatic Gripping: Lifting by One End
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究所（AIST））；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Ternopil Ivan Puluj National Technical University（捷尔诺波尔伊万·普卢伊国立技术大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Columbia University（哥伦比亚大学）；Toyota Research Institute（丰田研究院）
This paper introduces DextAIRity, an approach to manipulate deformable objects using active airflow. In contrast to conventional contact-based quasi-static manipulations, DextAIRity allows the system to apply dense forces on out-of-contact surfaces, expands the system's reach range, and provides safe high-speed interactions. These properties are particularly advantageous when manipulating under-actuated deformable objects with large surface areas or volumes. We demonstrate the effectiveness of DextAIRity through two challenging deformable object manipulation tasks: cloth unfolding and bag open…
DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）；Institute for Interdisciplinary Information Sciences (IIIS)（交叉信息研究院）
Learning Multimodal Attention for Manipulating Deformable Objects with Changing States
Future Robotics Organization, Waseda University（早稻田大学未来机器人组织）；Microsoft Research Asia（微软亚洲研究院）；Department of Modern Mechanical Engineering, Waseda University（早稻田大学现代机械工程系）；Artificial Intelligence Laboratories, Fujitsu Limited（Fujitsu 人工智能实验室）；Center for Technology Innovation - Controls and Robotics, Research & Development Group, Hitachi, Ltd.（富士通技术研发集团技术创新中心 - 控制与机器人）；Faculty of Science and Engineering, Waseda University（早稻田大学工学部）；National Institute of Advanced Science and Technology（国家先进科学研究院）
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）
Manipulating deformable linear objects (DLOs) to achieve desired shapes in constrained environments with obstacles is a meaningful but challenging task. Global planning is necessary for such a highly-constrained task; however, accurate models of DLOs required by planners are difficult to obtain owing to their deformable nature, and the inevitable modeling errors significantly affect the planning results, probably resulting in task failure if the robot simply executes the planned path in an open-loop manner. In this paper, we propose a coarse-to-fine framework to combine global planning and loc…
Hybrid Gripper with Passive Pneumatic Soft Joints for Grasping Deformable Thin Objects
School of Mechanical Engineering, Hanoi University of Science and Technology（河内科技大学机械工程学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）；Graduate School of Information Science and Technology, The University of Tokyo（东京大学信息科学与技术研究生院）
SCOPE: Smooth Convex Optimization for Planned Evolution of Deformable Linear Objects
Phystech School of Applied Mathematics and Computer Science（菲斯特克应用数学与计算机科学学院）；MIPT Russia Research Center for Artificial Intelligence（俄罗斯莫斯科国立交通大学人工智能研究中心）；Innopolis University（英诺普里斯大学）；Laboratory of Quantum Computing（量子计算实验室）；Center for Technologies in Robotics and Mechatronics Components（机器人与机电组件技术中心）
Manipulating deformable objects, such as ropes and clothing, is a long-standing challenge in robotics, because of their large degrees of freedom, complex non-linear dynamics, and self-occlusion in visual perception. The key difficulty is a suitable representation, rich enough to capture the object shape, dynamics for manipulation and yet simple enough to be estimated reliably from visual observations. This work aims to learn latent Graph dynamics for DefOrmable Object Manipulation (G-DOOM). G-DOOM approximates a deformable object as a sparse set of interacting keypoints, which are extracted au…
Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a pre-computed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and e…
We have seen much recent progress in rigid object manipulation, but interaction with deformable objects has notably lagged behind. Due to the large configuration space of deformable objects, solutions using traditional modelling approaches require significant engineering work. Perhaps then, bypassing the need for explicit modelling and instead learning the control in an end-to-end manner serves as a better approach? Despite the growing interest in the use of end-to-end robot learning approaches, only a small amount of work has focused on their applicability to deformable object manipulation. M…
AssistDLO: Assistive Teleoperation for Deformable Linear Object Manipulation
TU Darmstadt, Institute for Intelligent Autonomous Systems, Computer Science Department（德累斯顿技术大学智能自主系统研究所，计算机科学系）；Honda Research Institute Europe GmbH（本田欧洲研究院）；German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning（德国人工智能研究中心（DFKI），研究部门：机器人学习的系统AI；黑森AI；德国机器人研究所（RIG）；认知科学中心）；Hessian.AI；Robotics Institute Germany (RIG)；Centre for Cognitive Science
Prescribed Performance Control of Deformable Object Manipulation in Spatial Latent Space
Department of Mechanical Engineering, The Hong Kong Polytechnic University（机械工程系，香港理工大学）；T Stone Robotics Institute, Department of Mechanical and Automation Eng., The Chinese University of Hong Kong（T Stone机器人研究所，机械与自动化工程系，香港中文大学）
Certifiably Safe Manipulation of Deformable Linear Objects via Joint Shape and Tension Prediction
Department of Robotics, University of Michigan（机器人学系，密歇根大学）；Department of Mechanical Science and Engineering, University of Illinois at Urbana-Champaign（机械科学与工程系，伊利诺伊大学厄巴纳-香槟分校）
DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation
Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）；National University of Singapore（新加坡国立大学）；Technical University of Munich（慕尼黑工业大学）；NVIDIA（英伟达）
DFKI GmbH（德国人工智能研究中心）；Scuola Superiore Sant’Anna（圣安娜高等研究院）；University of Bremen（不来梅大学）
Manipulating objects with robotic hands is a complicated task. Not only the fingers of the hand, but also the pose of the robot's end effector need to be coordinated. Using human demonstrations of movements is an intuitive and data-efficient way of guiding the robot's behavior. We propose a modular framework with an automatic embodiment mapping to transfer recorded human hand motions to robotic systems. In this work, we use motion capture to record human motion. We evaluate our approach on eight challenging tasks, in which a robotic hand needs to grasp and manipulate either deformable or small…
Deformable object manipulation has many applications such as cooking and laundry folding in our daily lives. Manipulating elastoplastic objects such as dough is particularly challenging because dough lacks a compact state representation and requires contact-rich interactions. We consider the task of flattening a piece of dough into a specific shape from RGB-D images. While the task is seemingly intuitive for humans, there exist local optima for common approaches such as naive trajectory optimization. We propose a novel trajectory optimizer that optimizes through a differentiable "reset" module…
Robotic Control of the Deformation of Soft Linear Objects Using Deep Reinforcement Learning
CNRS（法国国家科学研究中心）；Clermont Auvergne INP（克莱蒙奥弗涅国立理工学院）；Institut Pascal（帕斯卡研究所）；Université Clermont Auvergne（克莱蒙奥弗涅大学）；Instituto de Investigación en Ingeniería de Aragón（阿拉贡工程研究所）；Universidad de Zaragoza（萨拉戈萨大学）；Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS)（智能技术卓越研究中心（CiTIUS））；Universidade de Santiago de Compostela（圣地亚哥德孔波斯特拉大学）
Autonomous dual-arm manipulation is an essential skill to deploy robots in unstructured scenarios. However, this is a challenging undertaking, particularly in terms of perception and planning. Unstructured scenarios are full of objects with different shapes and appearances that have to be grasped in a very specific manner so they can be functionally used. In this paper we present an integrated approach to perform dual-arm pick tasks autonomously. Our method consists of semantic segmentation, object pose estimation, deformable model registration, grasp planning and arm trajectory optimization. …
Robotic manipulation can be formulated as inducing a sequence of spatial displacements: where the space being moved can encompass an object, part of an object, or end effector. In this work, we propose the Transporter Network, a simple model architecture that rearranges deep features to infer spatial displacements from visual input - which can parameterize robot actions. It makes no assumptions of objectness (e.g. canonical poses, models, or keypoints), it exploits spatial symmetries, and is orders of magnitude more sample efficient than our benchmarked alternatives in learning vision-based ma…
The physical interaction of aerial robots with their environment has countless potential applications and is an emerging area with many open challenges. Fully-actuated multirotors have been introduced to tackle some of these challenges. They provide complete control over position and orientation and eliminate the need for attaching a multi-DoF manipulation arm to the robot. However, there are many open problems before they can be used in real-world applications. Researchers have introduced some methods for physical interaction in limited settings. Their experiments primarily use prototype-leve…
