The manipulation of deformable linear objects (DLOs) via model-based control requires an accurate and computationally efficient dynamics model. Yet, data-driven DLO dynamics models require large training data sets while their predictions often do not generalize, whereas physics-based models rely on good approximations of physical phenomena and often lack accuracy. To address these challenges, we propose a physics-informed neural ODE capable of predicting agile movements with significantly less data and hyper-parameter tuning. In particular, we model DLOs as serial chains of rigid bodies interconnected by passive elastic joints in which interaction forces are predicted by neural networks. The proposed model accurately predicts the motion of an robotically-actuated aluminium rod and an elastic foam cylinder after being trained on only thirty seconds of data.
The project code and data are available at: \url{https://tinyurl.com/neuralprba}
Robotic manipulation of deformable objects remains a challenging task. One such task is folding a garment autonomously. Given start and end folding positions, what is an optimal trajectory to move the robotic arm to fold a garment? Certain trajectories will cause the garment to move, creating wrinkles, and gaps, other trajectories will fail altogether. We present a novel solution to find an optimal trajectory that avoids such problematic scenarios. The trajectory is optimized by minimizing a quadratic objective function in an off-line simulator, which includes material properties of the garmen…
University of California, Berkeley（加州大学伯克利分校）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
This paper presents a novel trajectory optimization formulation to solve the robotic assembly of the belt drive unit. Robotic manipulations involving contacts and deformable objects are challenging in both dynamic modeling and trajectory planning. For modeling, variations in the belt tension and contact forces between the belt and the pulley could dramatically change the system dynamics. For trajectory planning, it is computationally expensive to plan trajectories for such hybrid dynamical systems as it usually requires planning for discrete modes separately. In this work, we formulate the bel…
Enhanced Robotic Navigation in Deformable Environments using Learning from Demonstration and Dynamic Modulation
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich (TUM), Germany（慕尼黑机器人与机器智能研究所（MIRMI），慕尼黑技术大学（TUM））；Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE（穆罕默德·本·扎耶德人工智能大学）
Tsinghua University（清华大学）；University of California, Berkeley（加州大学伯克利分校）
Point set registration is a powerful method that enables robots to manipulate deformable objects. By mapping the point cloud of the current object to the pre-trained point cloud, a transformation function can be constructed. The manipulator's trajectory for pre-trained shapes can be warped with this transformation function, yielding a feasible trajectory for the new shape. However, usually this transformation function regards objects as discrete points, and dismisses the topological structures. Therefore, it risks over-stretching or over-compression during manipulation. To tackle this problem,…
DeformGen: Dynamics-Based Topology Augmentation for Deformable Manipulation Policy Learning
Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）；Tsinghua University（清华大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Zhongguancun Academy（中关村学院）；Microsoft Research（微软研究院）
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables…
Deformable object manipulation requires computationally efficient representations that are compatible with robotic sensing modalities. In this paper, we present VIRDO:an implicit, multi-modal, and continuous representation for deformable-elastic objects. VIRDO operates directly on visual (point cloud) and tactile (reaction forces) modalities and learns rich latent embeddings of contact locations and forces to predict object deformations subject to external contacts.Here, we demonstrate VIRDOs ability to: i) produce high-fidelity cross-modal reconstructions with dense unsupervised correspondenc…
LIRMM - Université de Montpellier CNRS（法国国家科学研究中心蒙彼利埃大学信息、机器人、微电子学实验室）；Delft University of Technology（代尔夫特理工大学）；Honda Research Institute, Europe（本田欧洲研究所）；The Hong Kong Polytechnic University（香港理工大学）
This paper proposes a unified vision-based manipulation framework using image contours of deformable/rigid objects. Instead of using human-defined cues, the robot automatically learns the features from processed vision data. Our method simultaneously generates -- from the same data -- both, visual features and the interaction matrix that relates them to the robot control inputs. Extraction of the feature vector and control commands is done online and adaptively, with little data for initialization. The method allows the robot to manipulate an object without knowing whether it is rigid or defor…
Existing shape estimation methods for deformable object manipulation suffer from the drawbacks of being off-line, model dependent, noise-sensitive or occlusion-sensitive, and thus are not appropriate for manipulation tasks requiring high precision. In this paper, we present a real-time shape estimation approach for autonomous robotic manipulation of 3D deformable objects. Our method fulfills all the requirements necessary for the high-quality deformable object manipulation in terms of being real-time, model-free and robust to noise and occlusion. These advantages are accomplished using a joint…
Using a deformable sheet to handle objects is convenient and found in many practical applications. For object manipulation through a deformable sheet that is held by multiple mobile robots, it is a challenging task to model the object-sheet interactions. We present a computational model and algorithm to capture the object position on the deformable sheet with changing robotic team formations. A virtual variable cables model (VVCM) is proposed to simplify the modeling of the robot-sheet-object system. With the VVCM, we further present a motion planner for the robotic team to transport the objec…
Dexterous Manipulation of Deformable Objects via Pneumatic Gripping: Lifting by One End
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本国立先进工业科学技术研究所（AIST））；The University of Texas at Austin（得克萨斯大学奥斯汀分校）；Ternopil Ivan Puluj National Technical University（捷尔诺波尔伊万·普卢伊国立技术大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
In this paper, we presented a new method for deformation control of deformable objects, which utilizes both visual and tactile feedback. At present, manipulation of deformable objects is basically formulated by assuming positional constraints. But in fact, in many situations manipulation has to be performed under actively applied force constraints. This scenario is considered in this research. In the proposed scheme a tactile feedback is integrated to ensure a stable contact between the robot end-effector and the soft object to be manipulated. The controlled contact force is also utilized to r…
In this paper, we present a general approach to automatically visual-servo control the position and shape of a deformable object whose deformation parameters are unknown. The servo-control is achieved by online learning a model mapping between the robotic end-effector's movement and the object's deformation measurement. The model is learned using the Gaussian Process Regression (GPR) to deal with its highly nonlinear property, and once learned, the model is used for predicting the required control at each time step. To overcome GPR's high computational cost while dealing with long manipulation…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Humans can quickly determine the force required to grasp a deformable object to prevent its sliding or excessive deformation through vision and touch, which is still a challenging task for robots. To address this issue, we propose a novel 3D convolution-based visual-tactile fusion deep neural network (C3D-VTFN) to evaluate the grasp state of various deformable objects in this paper. Specifically, we divide the grasp states of deformable objects into three categories of sliding, appropriate and excessive. Also, a dataset for training and testing the proposed network is built by extensive graspi…
University of California, Berkeley（加州大学伯克利分校）；NVIDIA Corporation（英伟达公司）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
Robotic grasping of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, developing grasp strategies for such objects is uniquely challenging. Unlike rigid objects, deformable objects have infinite degrees of freedom and require field quantities (e.g., deformation, stress) to fully define their state. As these quantities are not easily accessible in the real world, we propose studying interaction with deformable objects through physics-based simulation…
The University of North Carolina（北卡罗来纳大学）；City University of Hong Kong（香港城市大学）
In this paper, we present a general learning-based framework to automatically visual-servo control the position and shape of a deformable object with unknown deformation parameters. The servo-control is accomplished by learning a feedback controller that determines the robotic end-effector's movement according to the deformable object's current status. This status encodes the object's deformation behavior by using a set of observed visual features, which are either manually designed or automatically extracted from the robot's sensor stream. A feedback control policy is then optimized to push t…
ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing
Robot Perception Laboratory, Department of Engineering, King’s College London（机器人感知实验室，工程系，伦敦国王学院）；Personal Robotics Laboratory, Department of Electrical and Electronic Engineering, Imperial College London（个人机器人实验室，电子与电气工程系，伦敦帝国学院）
Regulating contact forces with high precision is crucial for grasping and manipulating fragile or deformable objects. We aim to utilize the dexterity of human hands to regulate the contact forces for robotic hands and exploit human sensory-motor synergies in a wearable and non-invasive way. We extracted force information from the electric activities of skeletal muscles during their voluntary contractions through surface electromyography (sEMG). We built a regression model based on a Neural Network to predict the gripping force from the preprocessed sEMG signals and achieved high accuracy (R2 =…
Grasping deformable objects is not well researched due to the complexity in modelling and simulating the dynamic behavior of such objects. However, with the rapid development of physics-based simulators that support soft bodies, the research gap between rigid and deformable objects is getting smaller. To leverage the capability of such simulators and to challenge the assumption that has guided robotic grasping research so far, i.e., object rigidity, we proposed a deep-learning based approach that generates stiffness-dependent grasps. Our network is trained on purely synthetic data generated fr…
Cognitive Robotics, TU Delft（代尔夫特理工大学认知机器人）；Honda Research Institute, Europe（本田研究院欧洲）；LIRMM - Université de Montpellier CNRS（蒙彼利埃大学 CNRS LIRMM）；COSMER laboratory EA 7398, Universié de Toulon（土伦大学 COSMER 实验室 EA 7398）；The Hong Kong Polytechnic University（香港理工大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Michigan（密歇根大学）；Università degli Studi di Napoli Federico II（那不勒斯费德里科二世大学）；Osaka University（大阪大学）；National Institute of AIST（产业技术综合研究所）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Deformable object manipulation (DOM) is an emerging research problem in robotics. The ability to manipulate deformable objects endows robots with higher autonomy and promises new applications in the industrial, services, and healthcare sectors. However, compared to rigid object manipulation, the manipulation of deformable objects is considerably more complex, and is still an open research problem. Addressing DOM challenges demand breakthroughs in almost all aspects of robotics, namely hardware design, sensing, (deformation) modeling, planning, and control. In this article, we review recent adv…
KTH, Royal Institute of Technology（瑞典皇家理工学院）；CAS/RPL；Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人形与机器人研究所）
Capturing scene dynamics and predicting the future scene state is challenging but essential for robotic manipulation tasks, especially when the scene contains both rigid and deformable objects. In this work, we contribute a simulation environment and generate a novel dataset for task-specific manipulation, involving interactions between rigid objects and a deformable bag. The dataset incorporates a rich variety of scenarios including different object sizes, object numbers and manipulation actions. We approach dynamics learning by proposing an object-centric graph representation and two modules…
We present a framework for deformable object manipulation that interleaves planning and control, enabling complex manipulation tasks without relying on high-fidelity modeling or simulation. The key question we address is when should we use planning and when should we use control to achieve the task? Planners are designed to find paths through complex configuration spaces, but for highly underactuated systems, such as deformable objects, achieving a specific configuration is very difficult even with high-fidelity models. Conversely, controllers can be designed to achieve specific configurations…
Coordinated Manipulation of Hybrid Deformable-Rigid Objects in Constrained Environments
Department of Mechanical and Nuclear Engineering, Khalifa University（卡利法大学机械与核工程系）；Khalifa University Center for Autonomous Robotic Systems (KUCARS)（卡利法大学自主机器人系统中心）；Department of Computer Science, Khalifa University（卡利法大学计算机科学系）
Carnegie Mellon University（卡内基梅隆大学）；UC San Diego（加州大学圣地亚哥分校）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkil…
Controlling Deformable Objects with Non-negligible Dynamics: a Shape-Regulation Approach to End-Point Positioning
Department of Cognitive Robotics, Delft University of Technology（认知机器人系，代尔夫特理工大学）；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（机器人与机电系统研究所，德国航空航天中心（DLR））
DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）；Institute for Interdisciplinary Information Sciences (IIIS)（交叉信息研究院）
Estimating the state of a deformable object is crucial for robotic manipulation, yet accurate tracking is challenging when the object is partially-occluded. To address this problem, we propose an occlusion-robust RGBD sequence tracking framework based on Coherent Point Drift (CPD). To mitigate the effects of occlusion, our method 1) Uses a combination of locally linear embedding and constrained optimization to regularize the output of CPD, thus enforcing topological consistency when occlusions create disconnected pieces of the object; 2) Reasons about the free-space visible by an RGBD sensor t…
Sim-to-Real Gentle Manipulation of Deformable and Fragile Objects with Stress-Guided Reinforcement Learning
the division of Robotics, Perception and Learning, KTH Royal Institute of Technology（机器人、感知与学习部门，皇家理工学院）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（机器人与信息技术工业研究所）
Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor
School of Mechanical Engineering, Anhui University of Technology（安徽工业大学机械工程学院）；School of Engineering and Technology, China University of Geosciences (Beijing)（中国地质大学（北京）工程技术学院）
University of California, Berkeley（加州大学伯克利分校）；NVIDIA Corporation（英伟达公司）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
Robotic grasping of 3D deformable objects (e.g., fruits/vegetables, internal organs, bottles/boxes) is critical for real-world applications such as food processing, robotic surgery, and household automation. However, developing grasp strategies for such objects is uniquely challenging. In this work, we efficiently simulate grasps on a wide range of 3D deformable objects using a GPU-based implementation of the corotational finite element method (FEM). To facilitate future research, we open-source our simulated dataset (34 objects, 1e5 Pa elasticity range, 6800 grasp evaluations, 1.1M grasp meas…
Aalto University（阿尔托大学）；Technische Universität München（慕尼黑工业大学）
Planning robotic manipulation tasks, especially those that involve interaction between deformable and rigid objects, is challenging due to the complexity in predicting such interactions. We introduce SPONGE, a sequence planning pipeline powered by a deep learning-based contact prediction model for contacts between deformable and rigid bodies under interactions. The contact prediction model is trained on synthetic data generated by a developed simulation environment to learn the mapping from point-cloud observation of a rigid target object and the pose of a deformable tool, to 3D representation…
