Simulation modeling of robots, objects, and environments is the backbone for all model-based control and learning. It is leveraged broadly across dynamic programming and model-predictive control, as well as data generation for imitation, transfer, and reinforcement learning. In addition to fidelity, key features of models in these control and learning contexts are speed, stability, and native differentiability. However, many popular simulation platforms for robotics today lack at least one of the features above. More recently, position-based dynamics (PBD) has become a very popular simulation tool for modeling complex scenes of rigid and non-rigid object interactions, due to its speed and stability, and is starting to gain significant interest in robotics for its potential use in model-based control and learning. Thus, in this paper, we present a mathematical formulation for coupling position-based dynamics (PBD) simulation and optimal robot design, model-based motion control and system identification. Our framework breaks down PBD definitions and derivations for various types of joint-based articulated rigid bodies. We present a back-propagation method with automatic differentiation, which can integrate both positional and angular geometric constraints. Our framework can critically provide the native gradient information and perform gradient-based optimization tasks. We also propose articulated joint model representations and simulation workflow for our differentiable framework. We demonstrate the capability of the framework in efficient optimal robot design, accurate trajectory torque estimation and supporting spring stiffness estimation, where we achieve minor errors. We also implement impedance control in real robots to demonstrate the potential of our differentiable framework in human-in-the-loop applications.
Explosive Jumping with Rigid and Articulated Soft Quadrupeds via Example Guided Reinforcement Learning
Cognitive Robotics, Delft University of Technology（德鲁夫特理工大学认知机器人实验室）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Department of Industrial Engineering, University of Trento（特伦托大学工业工程系）；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（德国航空航天中心机器人与机电一体化研究所）
Versatile, Robust, and Explosive Locomotion with Rigid and Articulated Compliant Quadrupeds
Cognitive Robotics, Delft University of Technology（代尔夫特理工大学认知机器人学系）；University of Trento（特伦托大学）；University of Manchester（曼彻斯特大学）；Institute of Robotics and Mechatronics, German Aerospace Center (DLR)（德国航空航天中心机器人与机电一体化研究所）
University of Edinburgh（爱丁堡大学）；University of Oxford（牛津大学）
Haptic interaction is essential for the dynamic dexterity of animals, which seamlessly switch from an impedance to an admittance behaviour using the force feedback from their proprioception. However, this ability is extremely challenging to reproduce in robots, especially when dealing with complex interaction dynamics, distributed contacts, and contact switching. Current model-based controllers require accurate interaction modelling to account for contacts and stabilise the interaction. In this manuscript, we propose an adaptive force/position controller that exploits the fractal impedance con…
University of Maryland, College Park（马里兰大学帕克分校）；Intel Labs（英特尔实验室）
We present a method for efficient differentiable simulation of articulated bodies. This enables integration of articulated body dynamics into deep learning frameworks, and gradient-based optimization of neural networks that operate on articulated bodies. We derive the gradients of the forward dynamics using spatial algebra and the adjoint method. Our approach is an order of magnitude faster than autodiff tools. By only saving the initial states throughout the simulation process, our method reduces memory requirements by two orders of magnitude. We demonstrate the utility of efficient different…
ETH Zürich（苏黎世联邦理工学院）；Université de Montréal（蒙特利尔大学）；Disney Research（迪士尼研究院）
We present a differentiable dynamics solver that is able to handle frictional contact for rigid and deformable objects within a unified framework. Through a principled mollification of normal and tangential contact forces, our method circumvents the main difficulties inherent to the non-smooth nature of frictional contact. We combine this new contact model with fully-implicit time integration to obtain a robust and efficient dynamics solver that is analytically differentiable. In conjunction with adjoint sensitivity analysis, our formulation enables gradient-based optimization with adaptive tr…
We present a fast and feature-complete differentiable physics engine, Nimble (nimblephysics.org), that supports Lagrangian dynamics and hard contact constraints for articulated rigid body simulation. Our differentiable physics engine offers a complete set of features that are typically only available in non-differentiable physics simulators commonly used by robotics applications. We solve contact constraints precisely using linear complementarity problems (LCPs). We present efficient and novel analytical gradients through the LCP formulation of inelastic contact that exploit the sparsity of th…
Carnegie Mellon University（卡内基梅隆大学）；Australian National University（澳大利亚国立大学）
Dynamic Object-aware SLAM (DOS) exploits object-level information to enable robust motion estimation in dynamic environments. Existing methods mainly focus on identifying and excluding dynamic objects from the optimization. In this paper, we show that feature-based visual SLAM systems can also benefit from the presence of dynamic articulated objects by taking advantage of two observations: (1) The 3D structure of each rigid part of articulated object remains consistent over time; (2) The points on the same rigid part follow the same motion. In particular, we present AirDOS, a dynamic object-aw…
NVIDIA（英伟达）；Stanford University（斯坦福大学）；Microsoft Research（微软研究院）；Google Research（谷歌研究院）；University of Southern California（南加州大学）
Being able to reproduce physical phenomena ranging from light interaction to contact mechanics, simulators are becoming increasingly useful in more and more application domains where real-world interaction or labeled data are difficult to obtain. Despite recent progress, significant human effort is needed to configure simulators to accurately reproduce real-world behavior. We introduce a pipeline that combines inverse rendering with differentiable simulation to create digital twins of real-world articulated mechanisms from depth or RGB videos. Our approach automatically discovers joint types a…
University of Bordeaux（波尔多大学）；University Politehnica of Bucharest（布加勒斯特理工大学）
Attempts to install a rotating tool at the end of a robot arm poly-articulated date back twenty years, but these robots were not designed for that. Indeed, two essential features are necessary for machining: high rigidity and precision in a given workspace. The experimental results presented are the dynamic identification of a poly-articulated robot equipped with an integrated spindle. This study aims to highlight the influence of the geometric configuration of the robot arm on the overall stiffness of the system. The spindle is taken into account as an additional weight on board but also as a…
University of Maryland, College Park（马里兰大学帕克分校）；Intel Labs（英特尔研究院）
We present a method for differentiable simulation of soft articulated bodies. Our work enables the integration of differentiable physical dynamics into gradient-based pipelines. We develop a top-down matrix assembly algorithm within Projective Dynamics and derive a generalized dry friction model for soft continuum using a new matrix splitting strategy. We derive a differentiable control framework for soft articulated bodies driven by muscles, joint torques, or pneumatic tubes. The experiments demonstrate that our designs make soft body simulation more stable and realistic compared to other fra…
ArtGS:3D Gaussian Splatting for Interactive Visual-Physical Modeling and Manipulation of Articulated Objects
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；National University of Singapore（国立新加坡大学）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；Hefei University of Technology（合肥工业大学）
University of Michigan（密歇根大学）；Morehouse College（莫尔豪斯学院）
Articulated objects pose a unique challenge for robotic perception and manipulation. Their increased number of degrees-of-freedom makes tasks such as localization computationally difficult, while also making the process of real-world dataset collection unscalable. With the aim of addressing these scalability issues, we propose Neural Articulated Radiance Fields (NARF22), a pipeline which uses a fully-differentiable, configuration-parameterized Neural Radiance Field (NeRF) as a means of providing high quality renderings of articulated objects. NARF22 requires no explicit knowledge of the object…
Manipulating articulated objects requires multiple robot arms in general. It is challenging to enable multiple robot arms to collaboratively complete manipulation tasks on articulated objects. In this paper, we present $\textbf{V-MAO}$, a framework for learning multi-arm manipulation of articulated objects. Our framework includes a variational generative model that learns contact point distribution over object rigid parts for each robot arm. The training signal is obtained from interaction with the simulation environment which is enabled by planning and a novel formulation of object-centric co…
Many functional elements of human homes and workplaces consist of rigid components which are connected through one or more sliding or rotating linkages. Examples include doors and drawers of cabinets and appliances; laptops; and swivel office chairs. A robotic mobile manipulator would benefit from the ability to acquire kinematic models of such objects from observation. This paper describes a method by which a robot can acquire an object model by capturing depth imagery of the object as a human moves it through its range of motion. We envision that in future, a machine newly introduced to an e…
Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand
State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University（机械系统与振动国家重点实验室，机械工程学院，上海交通大学）；Shanghai Key Laboratory of Intelligent Robotics, Meta Robotics Institute, Shanghai Jiao Tong University（智能机器人上海重点实验室，元机器人研究院，上海交通大学）
University of Southern California（南加州大学）；Google（谷歌）
We present a differentiable simulation architecture for articulated rigid-body dynamics that enables the augmentation of analytical models with neural networks at any point of the computation. Through gradient-based optimization, identification of the simulation parameters and network weights is performed efficiently in preliminary experiments on a real-world dataset and in sim2sim transfer applications, while poor local optima are overcome through a random search approach.
Biology Laboratory, Technische Universit\" a t Berlin, Germany. denotes equal contribution.
We present a dataset with models of 14 articulated objects commonly found in human environments and with RGB-D video sequences and wrenches recorded of human interactions with them. The 358 interaction sequences total 67 minutes of human manipulation under varying experimental conditions (type of interaction, lighting, perspective, and background). Each interaction with an object is annotated with the ground truth poses of its rigid parts and the kinematic state obtained by a motion capture system. For a subset of 78 sequences (25 minutes), we also measured the interaction wrenches. The object…
Implicit Articulated Robot Morphology Modeling with Configuration Space Neural Signed Distance Functions
École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）；Chalmers University of Technology（查尔姆斯理工大学）；University College London（伦敦大学学院）；Rice University（莱斯大学）；Massachusetts Institute of Technology（麻省理工学院）
We propose an unsupervised vision-based system to estimate the joint configurations of the robot arm from a sequence of RGB or RGB-D images without knowing the model a priori, and then adapt it to the task of category-independent articulated object pose estimation. We combine a classical geometric formulation with deep learning and extend the use of epipolar constraint to multi-rigid-body systems to solve this task. Given a video sequence, the optical flow is estimated to get the pixel-wise dense correspondences. After that, the 6D pose is computed by a modified PnP algorithm. The key idea is …
CNRS-AIST JRL (Joint Robotics Laboratory) IRL（法国国家科学研究中心-产业技术综合研究所联合机器人实验室国际联合研究室）；National Institute of Advanced Industrial Science and Technology (AIST)（产业技术综合研究所）；University of Tsukuba（筑波大学）
For safe and reliable deployment of any robot controller on the real hardware platform, it is generally a necessary practice to comprehensively assess the performance of the controller with the specific robot in a realistic simulation environment beforehand. While there exist several software solutions that can provide the core physics engine for this purpose, it is often a cumbersome and error-prone effort to interface the simulation environment with the robot controller being evaluated. The controller may have a complex structure consisting of multiple states and transitions within a finite-…
We present Brax, an open source library for rigid body simulation with a focus on performance and parallelism on accelerators, written in JAX. We present results on a suite of tasks inspired by the existing reinforcement learning literature, but remade in our engine. Additionally, we provide reimplementations of PPO, SAC, ES, and direct policy optimization in JAX that compile alongside our environments, allowing the learning algorithm and the environment processing to occur on the same device, and to scale seamlessly on accelerators. Finally, we include notebooks that facilitate training of pe…
Soft actuators have drawn significant attention from researchers with an inherently compliant design to address the safety issues in physical human-robot interactions. However, they are also vulnerable and pose new challenges in the design, fabrication, and analysis due to their inherent material softness. In this paper, a novel hybrid actuator design is presented with bio-inspirations from the lobster, or crustaceans in a broader perspective. We enclose a soft chamber with rectangular cross-section using a series of articulated rigid shells to produce bending under pneumatic input. By mimicki…
Georgia Institute of Technology（佐治亚理工学院）；Stanford University（斯坦福大学）
We explore the problem of estimating the mass distribution of an articulated object by an interactive robotic agent. Our method predicts the mass distribution of an object by using the limited sensing and actuating capabilities of a robotic agent that is interacting with the object. We are inspired by the role of exploratory play in human infants. We take the combined approach of supervised and reinforcement learning to train an agent that learns to strategically interact with the object to estimate the object's mass distribution. Our method consists of two neural networks: (i) the policy netw…
Facebook AI Research（脸书人工智能研究院）；Intrinsic Innovation LLC.（Intrinsic创新有限责任公司）；Oxford Robotics Institute（牛津机器人研究所）
Building differentiable simulations of physical processes has recently received an increasing amount of attention. Specifically, some efforts develop differentiable robotic physics engines motivated by the computational benefits of merging rigid body simulations with modern differentiable machine learning libraries. Here, we present a library that focuses on the ability to combine data driven methods with analytical rigid body computations. More concretely, our library \emph{Differentiable Robot Models} implements both \emph{differentiable} and \emph{learnable} models of the kinematics and dyn…
We propose an algorithm to compute the dynamics of articulated rigid-bodies with different sensor distributions. Prior to the on-line computations, the proposed algorithm performs an off-line optimisation step to simplify the computational complexity of the underlying solution. This optimisation step consists in formulating the dynamic computations as a system of linear equations. The computational complexity of computing the associated solution is reduced by performing a permuted LU-factorisation with off-line optimised permutations. We apply our algorithm to solve classical dynamic problems:…
FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenario
CFCS, School of Computer Science, Peking University（计算机科学系，北京大学CFCS）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；PrimeBot
City University of Hong Kong（香港城市大学）；Zhejiang University of Technology（浙江工业大学）
Motion behaviors of a rigid body can be characterized by a 6-dimensional motion trajectory, which contains position vectors of a reference point on the rigid body and rotations of this rigid body over time. This paper devises a Rotation and Relative Velocity (RRV) descriptor by exploring the local translational and rotational invariants of motion trajectories of rigid bodies, which is insensitive to noise, invariant to rigid transformation and scaling. A flexible metric is also introduced to measure the distance between two RRV descriptors. The RRV descriptor is then applied to characterize mo…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Differentiable physics is a powerful tool in computer vision and robotics for scene understanding and reasoning about interactions. Existing approaches have frequently been limited to objects with simple shape or shapes that are known in advance. In this paper, we propose a novel approach to differentiable physics with frictional contacts which represents object shapes implicitly using signed distance fields (SDFs). Our simulation supports contact point calculation even when the involved shapes are nonconvex. Moreover, we propose ways for differentiating the dynamics for the object shape to fa…
We present a differentiable formulation of rigid-body contact dynamics for objects and robots represented as compositions of convex primitives. Existing optimization-based approaches simulating contact between convex primitives rely on a bilevel formulation that separates collision detection and contact simulation. These approaches are unreliable in realistic contact simulation scenarios because isolating the collision detection problem introduces contact location non-uniqueness. Our approach combines contact simulation and collision detection into a unified single-level optimization problem. …
