We propose to use a simulation driven inverse inference approach to model the dynamics of tree branches under manipulation. Learning branch dynamics and gaining the ability to manipulate deformable vegetation can help with occlusion-prone tasks, such as fruit picking in dense foliage, as well as moving overhanging vines and branches for navigation in dense vegetation. The underlying deformable tree geometry is encapsulated as coarse spring abstractions executed on parallel, non-differentiable simulators. The implicit statistical model defined by the simulator, reference trajectories obtained by actively probing the ground truth, and the Bayesian formalism, together guide the spring parameter posterior density estimation. Our non-parametric inference algorithm, based on Stein Variational Gradient Descent, incorporates biologically motivated assumptions into the inference process as neural network driven learnt joint priors; moreover, it leverages the finite difference scheme for gradient approximations. Real and simulated experiments confirm that our model can predict deformation trajectories, quantify the estimation uncertainty, and it can perform better when base-lined against other inference algorithms, particularly from the Monte Carlo family. The model displays strong robustness properties in the presence of heteroscedastic sensor noise; furthermore, it can generalise to unseen grasp locations.
Manipulation of granular materials such as sand or rice remains an unsolved problem due to challenges such as the difficulty of defining their configuration or modeling the materials and their particles interactions. Current approaches tend to simplify the material dynamics and omit the interactions between the particles. In this paper, we propose to use a graph-based representation to model the interaction dynamics of the material and rigid bodies manipulating it. This allows the planning of manipulation trajectories to reach a desired configuration of the material. We use a graph neural netw…
Virginia Tech（弗吉尼亚理工大学）；University of Genova（热那亚大学）
In this article, we propose a new approach for simulating trees, including their branches, sub-branches, and leaves. This approach combines the theory of biological development, mathematical models, and computer graphics, producing simulated trees and forest with full geometry. Specifically, we adopt the Lindenmayer process to simulate the branching pattern of trees and modify the available measurements and dimensions of 3D CAD developed object files to create natural looking sub-branches and leaves. Randomization has been added to the placement of all branches, sub branches and leaves. To sim…
AMBER: A tether-deployable gripping crawler with compliant microspines for canopy manipulation
eAviation Laboratory, TUM School of Engineering and Design（eAviation实验室，技术大学工程与设计学院）；Laboratory of Sustainability Robotics, EMPA（可持续机器人实验室，EMPA）；Laboratory of Sustainability Robotics, EPFL（可持续机器人实验室，瑞士联邦理工学院）；Aerial Robotics Laboratory, Imperial College London（空域机器人实验室，伦敦帝国理工学院）；Bristol University（布里斯托大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；University of Salerno（萨莱诺大学）；University of Cassino and Southern Lazio（卡西诺大学和南拉齐奥大学）
We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the …
For tasks where the dynamics of multiple agents are physically coupled, e.g., in cooperative manipulation, the coordination between the individual agents becomes crucial, which requires exact knowledge of the interaction dynamics. This problem is typically addressed using centralized estimators, which can negatively impact the flexibility and robustness of the overall system. To overcome this shortcoming, we propose a novel distributed learning framework for the exemplary task of cooperative manipulation using Bayesian principles. Using only local state information each agent obtains an estima…
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University（东南大学自动化学院）；Key Laboratory of Measurement and Control of CSE, Ministry of Education（东南大学复杂工程系统测量与控制教育部重点实验室）；Department of Engineering, King’s College London（伦敦国王学院工程系）
Northeastern University（东北大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）
We present a deep imitation learning framework for robotic bimanual manipulation in a continuous state-action space. A core challenge is to generalize the manipulation skills to objects in different locations. We hypothesize that modeling the relational information in the environment can significantly improve generalization. To achieve this, we propose to (i) decompose the multi-modal dynamics into elemental movement primitives, (ii) parameterize each primitive using a recurrent graph neural network to capture interactions, and (iii) integrate a high-level planner that composes primitives sequ…
Stanford University（斯坦福大学）；NVIDIA（英伟达）；California Institute of Technology（加州理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through intelligent choice of action space. Operational Space Control (OSC) has been used as an effective task-space controller for manipulation. Nonetheless, its strength depends on the underlying modeling fidelity, and is prone to failure when there are modeling errors. In this work, we propose OSC for Adaptation and Robustness (OSCAR), a data-driven variant of OSC that compensates for modeling errors by inferring relevant dy…
In this paper, we propose a data-driven skill learning approach to solve highly dynamic manipulation tasks entirely from offline teleoperated play data. We use a bilateral teleoperation system to continuously collect a large set of dexterous and agile manipulation behaviors, which is enabled by providing direct force feedback to the operator. We jointly learn the state conditional latent skill distribution and skill decoder network in the form of goal-conditioned policy and skill conditional state transition dynamics using a two-stage generative modeling framework. This allows one to perform r…
Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model
University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Tsinghua University（清华大学）；JD Explore Academy（京东探索学院）；ShanghaiTech University（上海科技大学）；Nanjing University（南京大学）
Robotic manipulation of deformable objects is a difficult problem especially because of the complexity of the many different ways an object can deform. Searching such a high dimensional state space makes it difficult to recognize, track, and manipulate deformable objects. In this paper, we introduce a predictive, model-driven approach to address this challenge, using a pre-computed, simulated database of deformable object models. Mesh models of common deformable garments are simulated with the garments picked up in multiple different poses under gravity, and stored in a database for fast and e…
Improving Robustness to Out-of-Distribution States in Imitation Learning via Deep Koopman-Boosted Diffusion Policy
Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich (TUM)（计算机辅助医疗程序与增强现实中心（CAMP），慕尼黑技术大学（TUM））；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Medical Intelligence and Robotic Cognition Lab (MIRoC), The University of Hong Kong (HKU)（医疗智能与机器人认知实验室（MIRoC），香港大学（HKU））
University of Edinburgh（爱丁堡大学）；Heriot-Watt University（赫瑞-瓦特大学）
In an attempt to confer robots with complex manipulation capabilities, dual-arm anthropomorphic systems have become an important research topic in the robotics community. Most approaches in the literature rely upon a great understanding of the dynamics underlying the system's behaviour and yet offer limited autonomous generalisation capabilities. To address these limitations, this work proposes a modelisation for dual-arm manipulators based on dynamic movement primitives laying in two orthogonal spaces. The modularity and learning capabilities of this model are leveraged to formulate a novel e…
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further prop…
