In this paper, we present a general approach to automatically visual-servo control the position and shape of a deformable object whose deformation parameters are unknown. The servo-control is achieved by online learning a model mapping between the robotic end-effector's movement and the object's deformation measurement. The model is learned using the Gaussian Process Regression (GPR) to deal with its highly nonlinear property, and once learned, the model is used for predicting the required control at each time step. To overcome GPR's high computational cost while dealing with long manipulation sequences, we implement a fast online GPR by selectively removing uninformative observation data from the regression process. We validate the performance of our controller on a set of deformable object manipulation tasks and demonstrate that our method can achieve effective and accurate servo-control for general deformable objects with a wide variety of goal settings. Experiment videos are available at https://sites.google.com/view/mso-fogpr
Existing shape estimation methods for deformable object manipulation suffer from the drawbacks of being off-line, model dependent, noise-sensitive or occlusion-sensitive, and thus are not appropriate for manipulation tasks requiring high precision. In this paper, we present a real-time shape estimation approach for autonomous robotic manipulation of 3D deformable objects. Our method fulfills all the requirements necessary for the high-quality deformable object manipulation in terms of being real-time, model-free and robust to noise and occlusion. These advantages are accomplished using a joint…
LIRMM - Université de Montpellier CNRS（法国国家科学研究中心蒙彼利埃大学信息、机器人、微电子学实验室）；Delft University of Technology（代尔夫特理工大学）；Honda Research Institute, Europe（本田欧洲研究所）；The Hong Kong Polytechnic University（香港理工大学）
This paper proposes a unified vision-based manipulation framework using image contours of deformable/rigid objects. Instead of using human-defined cues, the robot automatically learns the features from processed vision data. Our method simultaneously generates -- from the same data -- both, visual features and the interaction matrix that relates them to the robot control inputs. Extraction of the feature vector and control commands is done online and adaptively, with little data for initialization. The method allows the robot to manipulate an object without knowing whether it is rigid or defor…
Columbia University（哥伦比亚大学）；Toyota Research Institute（丰田研究院）
This paper introduces DextAIRity, an approach to manipulate deformable objects using active airflow. In contrast to conventional contact-based quasi-static manipulations, DextAIRity allows the system to apply dense forces on out-of-contact surfaces, expands the system's reach range, and provides safe high-speed interactions. These properties are particularly advantageous when manipulating under-actuated deformable objects with large surface areas or volumes. We demonstrate the effectiveness of DextAIRity through two challenging deformable object manipulation tasks: cloth unfolding and bag open…
Cognitive Robotics, TU Delft（代尔夫特理工大学认知机器人）；Honda Research Institute, Europe（本田研究院欧洲）；LIRMM - Université de Montpellier CNRS（蒙彼利埃大学 CNRS LIRMM）；COSMER laboratory EA 7398, Universié de Toulon（土伦大学 COSMER 实验室 EA 7398）；The Hong Kong Polytechnic University（香港理工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Michigan（密歇根大学）；Università degli Studi di Napoli Federico II（那不勒斯费德里科二世大学）；Osaka University（大阪大学）；National Institute of AIST（产业技术综合研究所）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications in the industrial, services, and healthcare sectors. However, compared to rigid object manipulation, the manipulation of deformable objects is considerably more complex, and is still an open research problem. Addressing DOM challenges demand breakthroughs in almost all aspects of robotics, namely hardware design, sensing, (deformation) modeling, planning, and control. In this article, we review recent adv…
DeformGen: Dynamics-Based Topology Augmentation for Deformable Manipulation Policy Learning
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）；Tsinghua University（清华大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Zhongguancun Academy（中关村学院）；Microsoft Research（微软研究院）
The University of North Carolina（北卡罗来纳大学）；City University of Hong Kong（香港城市大学）
In this paper, we present a general learning-based framework to automatically visual-servo control the position and shape of a deformable object with unknown deformation parameters. The servo-control is accomplished by learning a feedback controller that determines the robotic end-effector's movement according to the deformable object's current status. This status encodes the object's deformation behavior by using a set of observed visual features, which are either manually designed or automatically extracted from the robot's sensor stream. A feedback control policy is then optimized to push t…
Dexterous Manipulation of Deformable Objects via Pneumatic Gripping: Lifting by One End
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究所（AIST））；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Ternopil Ivan Puluj National Technical University（捷尔诺波尔伊万·普卢伊国立技术大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；City University of Hong Kong（香港城市大学）
The complex physical properties of highly deformable materials such as clothes pose significant challenges fanipulation systems. We present a novel visual feedback dictionary-based method for manipulating defoor autonomous robotic mrmable objects towards a desired configuration. Our approach is based on visual servoing and we use an efficient technique to extract key features from the RGB sensor stream in the form of a histogram of deformable model features. These histogram features serve as high-level representations of the state of the deformable material. Next, we collect manipulation data …
Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）
The deformable linear objects (DLOs) are common in both industrial and domestic applications, such as wires, cables, ropes. Because of its highly deformable nature, it is difficult for the robot to reproduce human's dexterous skills on DLOs. In this paper, the unknown deformation model is estimated in both the offline and online manners. The offline learning aims to provide a good approximation prior to the manipulation task, while the online learning aims to compensate the errors due to insufficient training (e.g. limited datasets) in the offline phase. The offline module works by constructin…
Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor
School of Mechanical Engineering, Anhui University of Technology（安徽工业大学机械工程学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）
Coordinated Manipulation of Hybrid Deformable-Rigid Objects in Constrained Environments
Department of Mechanical and Nuclear Engineering, Khalifa University（卡利法大学机械与核工程系）；Khalifa University Center for Autonomous Robotic Systems (KUCARS)（卡利法大学自主机器人系统中心）；Department of Computer Science, Khalifa University（卡利法大学计算机科学系）
University of California, Berkeley（加州大学伯克利分校）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
This paper presents a novel trajectory optimization formulation to solve the robotic assembly of the belt drive unit. Robotic manipulations involving contacts and deformable objects are challenging in both dynamic modeling and trajectory planning. For modeling, variations in the belt tension and contact forces between the belt and the pulley could dramatically change the system dynamics. For trajectory planning, it is computationally expensive to plan trajectories for such hybrid dynamical systems as it usually requires planning for discrete modes separately. In this work, we formulate the bel…
National University of Singapore（新加坡国立大学）；Sea AI Lab（Sea人工智能实验室）
Deformable Object Manipulation (DOM) is of significant importance to both daily and industrial applications. Recent successes in differentiable physics simulators allow learning algorithms to train a policy with analytic gradients through environment dynamics, which significantly facilitates the development of DOM algorithms. However, existing DOM benchmarks are either single-object-based or non-differentiable. This leaves the questions of 1) how a task-specific algorithm performs on other tasks and 2) how a differentiable-physics-based algorithm compares with the non-differentiable ones in ge…
Sashimi-Bot: Autonomous Tri-manual Advanced Manipulation and Cutting of Deformable Objects
Robotics and Intelligent Automation（机器人与智能自动化）；SINTEF Ocean（SINTEF海洋）；Department of Computer Science（计算机科学系）；NTNU - Norwegian University of Science and Technology（挪威科学与技术大学）；QUT Centre for Robotics（昆士兰大学机器人中心）；Queensland University of Technology（昆士兰技术大学）；Inria, University of Rennes, CNRS, IRISA（Inria、里昂大学、法国国家科学研究中心、IRISA）；CSAIL（计算机科学与人工智能实验室）；MIT - Massachusets Institute of Technology（麻省理工学院）；RealTek；Norwegian University of Life Sciences (NMBU)（挪威生命科学大学（NMBU））
Sim-to-Real Gentle Manipulation of Deformable and Fragile Objects with Stress-Guided Reinforcement Learning
the division of Robotics, Perception and Learning, KTH Royal Institute of Technology（机器人、感知与学习部门，皇家理工学院）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（机器人与信息技术工业研究所）
Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects
CSIRO Robotics, Data61, CSIRO（CSIRO机器人部、Data61、CSIRO）；School of Electrical Engineering and Computer Science (EECS), The University of Queensland（电气工程与计算机科学学院（EECS）、昆士兰大学）
DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）；Institute for Interdisciplinary Information Sciences (IIIS)（交叉信息研究院）
Robot manipulation of rope-like objects is an interesting problem that has some critical applications, such as autonomous robotic suturing. Solving for and controlling rope is difficult due to the complexity of rope physics and the challenge of building fast and accurate models of deformable materials. While more data-driven approaches have become more popular for finding controllers that learn to do a single task, there is still a strong motivation for a model-based method that could be used to solve a large variety of optimization problems. Towards this end, we introduced compliant, position…
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）
Point set registration is a powerful method that enables robots to manipulate deformable objects. By mapping the point cloud of the current object to the pre-trained point cloud, a transformation function can be constructed. The manipulator's trajectory for pre-trained shapes can be warped with this transformation function, yielding a feasible trajectory for the new shape. However, usually this transformation function regards objects as discrete points, and dismisses the topological structures. Therefore, it risks over-stretching or over-compression during manipulation. To tackle this problem,…
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）
Manipulating deformable linear objects (DLOs) to achieve desired shapes in constrained environments with obstacles is a meaningful but challenging task. Global planning is necessary for such a highly-constrained task; however, accurate models of DLOs required by planners are difficult to obtain owing to their deformable nature, and the inevitable modeling errors significantly affect the planning results, probably resulting in task failure if the robot simply executes the planned path in an open-loop manner. In this paper, we propose a coarse-to-fine framework to combine global planning and loc…
Columbia University（哥伦比亚大学）；Toyota Research Institute（丰田研究院）
This paper tackles the task of goal-conditioned dynamic manipulation of deformable objects. This task is highly challenging due to its complex dynamics (introduced by object deformation and high-speed action) and strict task requirements (defined by a precise goal specification). To address these challenges, we present Iterative Residual Policy (IRP), a general learning framework applicable to repeatable tasks with complex dynamics. IRP learns an implicit policy via delta dynamics -- instead of modeling the entire dynamical system and inferring actions from that model, IRP learns delta dynamic…
Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息学研究所）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Benchmarking of robotic manipulations is one of the open issues in robotic research. An important factor that has enabled progress in this area in the last decade is the existence of common object sets that have been shared among different research groups. However, the existing object sets are very limited when it comes to cloth-like objects that have unique particularities and challenges. This paper is a first step towards the design of a cloth object set to be distributed among research groups from the robotics cloth manipulation community. We present a set of household cloth objects and rel…
Carnegie Mellon University（卡内基梅隆大学）；Stanford University（斯坦福大学）；NVIDIA（英伟达）；University of Washington（华盛顿大学）；The University of Sydney（悉尼大学）
Deformable object manipulation remains a challenging task in robotics research. Conventional techniques for parameter inference and state estimation typically rely on a precise definition of the state space and its dynamics. While this is appropriate for rigid objects and robot states, it is challenging to define the state space of a deformable object and how it evolves in time. In this work, we pose the problem of inferring physical parameters of deformable objects as a probabilistic inference task defined with a simulator. We propose a novel methodology for extracting state information from …
With the field of rigid-body robotics having matured in the last fifty years, routing, planning, and manipulation of deformable objects have recently emerged as a more untouched research area in many fields ranging from surgical robotics to industrial assembly and construction. Routing approaches for deformable objects which rely on learned implicit spatial representations (e.g., Learning-from-Demonstration methods) make them vulnerable to changes in the environment and the specific setup. On the other hand, algorithms that entirely separate the spatial representation of the deformable object …
