Robotic manipulation of volumetric elastoplastic deformable materials, from foods such as dough to construction materials like clay, is in its infancy, largely due to the difficulty of modelling and perception in a high-dimensional space. Simulating the dynamics of such materials is computationally expensive. It tends to suffer from inaccurately estimated physics parameters of the materials and the environment, impeding high-precision manipulation. Estimating such parameters from raw point clouds captured by optical cameras suffers further from heavy occlusions. To address this challenge, this work introduces a novel Differentiable Physics-based System Identification (DPSI) framework that enables a robot arm to infer the physics parameters of elastoplastic materials and the environment using simple manipulation motions and incomplete 3D point clouds, aligning the simulation with the real world. Extensive experiments show that with only a single real-world interaction, the estimated parameters, Young's modulus, Poisson's ratio, yield stress and friction coefficients, can accurately simulate visually and physically realistic deformation behaviours induced by unseen and long-horizon manipulation motions. Additionally, the DPSI framework inherently provides physically intuitive interpretations for the parameters in contrast to black-box approaches such as deep neural networks. The project is fully open-sourced via https://ianyangchina.github.io/SI4RP-data/.
FORM: Robot Manipulation through Direct Material Law Identification
University of California, Berkeley（加州大学伯克利分校）；Chandra Family Department of Electrical and Computer Engineering（Chandra Family 电气与计算机工程系）；Maseeh Department of Civil, Architectural and Environmental Engineering（Maseeh 土木、建筑与环境工程系）；Oden Institute for Computational Engineering and Sciences（Oden 计算工程与科学研究所）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Diff-Transfer: Model-based Robotic Manipulation Skill Transfer via Differentiable Physics Simulation
Nanjing University（南京大学）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）
DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model
University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Sashimi-Bot: Autonomous Tri-manual Advanced Manipulation and Cutting of Deformable Objects
Robotics and Intelligent Automation（机器人与智能自动化）；SINTEF Ocean（SINTEF海洋）；Department of Computer Science（计算机科学系）；NTNU - Norwegian University of Science and Technology（挪威科学与技术大学）；QUT Centre for Robotics（昆士兰大学机器人中心）；Queensland University of Technology（昆士兰技术大学）；Inria, University of Rennes, CNRS, IRISA（Inria、里昂大学、法国国家科学研究中心、IRISA）；CSAIL（计算机科学与人工智能实验室）；MIT - Massachusets Institute of Technology（麻省理工学院）；RealTek；Norwegian University of Life Sciences (NMBU)（挪威生命科学大学（NMBU））
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；National Engineering Research Center for Visual Information and Applications（视觉信息与应用国家工程研究中心）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）
DiSECt: A Differentiable Simulator for Parameter Inference and Control in Robotic Cutting
University of Southern California（南加州大学）；NVIDIA（英伟达）；University of Washington（华盛顿大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Sydney（悉尼大学）
DiPGrasp: Parallel Local Searching for Efficient Differentiable Grasp Planning
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（上海交通大学电子信息与电气工程学院）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）
Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning
Robotics and Autonomous Systems Thrust, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）机器人与自主系统方向）；Institute of Data Science, The University of Hong Kong（香港大学数据科学学院）
Dextrous Tactile In-Hand Manipulation Using a Modular Reinforcement Learning Architecture
DLR Institute of Robotics and Mechatronics（DLR机器人与 mechatronics研究所）；Technical University of Munich（慕尼黑工业大学）；Deggendorf Institute of Technology（代根多夫应用技术大学）
MSGField: A Unified Scene Representation Integrating Motion, Semantics, and Geometry for Robotic Manipulation
School of Computer Science and Technology, University of Science and Technology of China (USTC)（中国科学技术大学计算机科学与技术学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Tongji University（同济大学）；Shanghai Research Institute for Intelligent Autonomous Systems（上海自主智能无人系统研究院）；Purdue University（普渡大学）；Nanyang Technological University, Singapore（新加坡南洋理工大学）
DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
University of California, San Diego（加州大学圣地亚哥分校）；Tsinghua University（清华大学）；Autodesk（欧特克）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
