SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation
Fudan University, China（复旦大学）；Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）；Shanghai Jiao Tong University, China（上海交通大学）；The Chinese University of Hong Kong, China（香港中文大学）；The University of Hong Kong, China（香港大学）
Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang
Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.
A task-sequencing simulator in robotics manipulation to integrate simulation-for-learning and simulation-for-execution is introduced. Unlike existing machine-learning simulation where a non-decomposed simulation is used to simulate a training scenario, the task-sequencing simulator runs a composed simulation using building blocks. This way, the simulation-for-learning is structured similarly to a multi-step simulation-for-execution. To compose both learning and execution scenarios, a unified trainable-and-composable description of blocks called a concept model is proposed and used. Using the s…
A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AI
Department of Computer Science, City University of Hong Kong（城市大学计算机科学系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Department of Informatics, Universität Hamburg（汉堡大学信息学院）
Queensland University of Technology (QUT)（昆士兰科技大学）；Data61/CSIRO（Data61/联邦科学与工业研究组织）；Australian Centre for Robotic Vision (ACRV)（澳大利亚机器人视觉中心）
We quantify the accuracy of various simulators compared to a real world robotic reaching and interaction task. Simulators are used in robotics to design solutions for real world hardware without the need for physical access. The `reality gap' prevents solutions developed or learnt in simulation from performing well, or at at all, when transferred to real-world hardware. Making use of a Kinova robotic manipulator and a motion capture system, we record a ground truth enabling comparisons with various simulators, and present quantitative data for various manipulation-oriented robotic tasks. We sh…
University of Hong Kong（香港大学）；Centre for Garment Production Limited（制衣生产中心有限公司）
Dynamic state representation learning is an important task in robot learning. Latent space that can capture dynamics related information has wide application in areas such as accelerating model free reinforcement learning, closing the simulation to reality gap, as well as reducing the motion planning complexity. However, current dynamic state representation learning methods scale poorly on complex dynamic systems such as deformable objects, and cannot directly embed well defined simulation function into the training pipeline. We propose DiffSRL, a dynamic state representation learning pipeline…
Carnegie Mellon University（卡内基梅隆大学）；UC San Diego（加州大学圣地亚哥分校）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
We consider the problem of sequential robotic manipulation of deformable objects using tools. Previous works have shown that differentiable physics simulators provide gradients to the environment state and help trajectory optimization to converge orders of magnitude faster than model-free reinforcement learning algorithms for deformable object manipulation. However, such gradient-based trajectory optimization typically requires access to the full simulator states and can only solve short-horizon, single-skill tasks due to local optima. In this work, we propose a novel framework, named DiffSkil…
WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；EvoPhys AI（EvoPhys人工智能公司）；Joy Future Academy, JD（京东探索研究院）；The University of Sydney（悉尼大学）；The Hong Kong University of Science and Technology（香港科技大学）；Beijing Institute of Technology（北京理工大学）
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University（东南大学自动化学院）；Key Laboratory of Measurement and Control of CSE, Ministry of Education（东南大学复杂工程系统测量与控制教育部重点实验室）；Department of Engineering, King’s College London（伦敦国王学院工程系）
Robot manipulation in cluttered scenes often requires contact-rich interactions with objects. It can be more economical to interact via non-prehensile actions, for example, push through other objects to get to the desired grasp pose, instead of deliberate prehensile rearrangement of the scene. For each object in a scene, depending on its properties, the robot may or may not be allowed to make contact with, tilt, or topple it. To ensure that these constraints are satisfied during non-prehensile interactions, a planner can query a physics-based simulator to evaluate the complex multi-body intera…
Embodied Tree of Thoughts: Deliberate Manipulation Planning with Embodied World Model
University of Chinese Academy of Sciences (UCAS)（中国科学院大学）；Tsinghua University（清华大学）；JD Explore Academy（京东探索学院）；ShanghaiTech University（上海科技大学）；Nanjing University（南京大学）
WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
The Hong Kong University of Science and Technology（香港科技大学）；Joy Future Academy（京东探索研究院）；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）
DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning
Imprintx Robotics（印迹机器人）；Electric Power Research Institute of State Grid Jibei Electric Power Co., Ltd.（国网冀北电力有限公司电力科学研究院）；North China Electric Power Research Institute Co., Ltd.（华北电力科学研究院有限责任公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
In-hand pivoting is one of the important manipulation skills that leverage robot grippers' extrinsic dexterity to perform repositioning tasks to compensate for environmental uncertainties and imprecise motion execution. Although many researchers have been trying to solve pivoting problems using mathematical modeling or learning-based approaches, the problems remain as open challenges. On the other hand, humans perform in-hand manipulation with remarkable precision and speed. Hence, the solution could be provided by making full use of this intrinsic human skill through dexterous teleoperation. …
Coupled Local and Global World Models for Efficient First Order RL
Machines in Motion Laboratory, New York University, USA（纽约大学运动机器实验室）；LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France（图卢兹大学LAAS-CNRS中心）；Artificial and Natural Intelligence Toulouse Institute, Toulouse, France（图卢兹人工智能与自然智能研究所）
Tandon School of Engineering, New York University（纽约大学坦登工程学院）
This paper presents an efficient approach to object manipulation planning using Monte Carlo Tree Search (MCTS) to find contact sequences and an efficient ADMM-based trajectory optimization algorithm to evaluate the dynamic feasibility of candidate contact sequences. To accelerate MCTS, we propose a methodology to learn a goal-conditioned policy-value network to direct the search towards promising nodes. Further, manipulation-specific heuristics enable to drastically reduce the search space. Systematic object manipulation experiments in a physics simulator and on real hardware demonstrate the e…
World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；Beijing Zhongke Huiling Robot Technology Co（北京中科创联机器人技术有限公司）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
University of California, Santa Cruz（加州大学圣克鲁兹分校）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robo…
University of Southern California（南加州大学）；NVIDIA（英伟达）；University of Washington（华盛顿大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Sydney（悉尼大学）
Robotic cutting of soft materials is critical for applications such as food processing, household automation, and surgical manipulation. As in other areas of robotics, simulators can facilitate controller verification, policy learning, and dataset generation. Moreover, differentiable simulators can enable gradient-based optimization, which is invaluable for calibrating simulation parameters and optimizing controllers. In this work, we present DiSECt: the first differentiable simulator for cutting soft materials. The simulator augments the finite element method (FEM) with a continuous contact m…
University of Edinburgh（爱丁堡大学）；Ghent University（根特大学）；imec（微电子研究中心）
Soft actuators offer a safe, adaptable approach to tasks like gentle grasping and dexterous manipulation. Creating accurate models to control such systems however is challenging due to the complex physics of deformable materials. Accurate Finite Element Method (FEM) models incur prohibitive computational complexity for closed-loop use. Using a differentiable simulator is an attractive alternative, but their applicability to soft actuators and deformable materials remains underexplored. This paper presents a framework that combines the advantages of both. We learn a differentiable model consist…
We propose to perform imitation learning for dexterous manipulation with multi-finger robot hand from human demonstrations, and transfer the policy to the real robot hand. We introduce a novel single-camera teleoperation system to collect the 3D demonstrations efficiently with only an iPad and a computer. One key contribution of our system is that we construct a customized robot hand for each user in the physical simulator, which is a manipulator resembling the same kinematics structure and shape of the operator's hand. This provides an intuitive interface and avoid unstable human-robot hand r…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；TU Berlin（柏林工业大学）；University Stuttgart（斯图加特大学）
Physical reasoning is a core aspect of intelligence in animals and humans. A central question is what model should be used as a basis for reasoning. Existing work considered models ranging from intuitive physics and physical simulators to contact dynamics models used in robotic manipulation and locomotion. In this work we propose descriptions of physics which directly allow us to leverage optimization methods for physical reasoning and sequential manipulation planning. The proposed multi-physics formulation enables the solver to mix various levels of abstraction and simplifications for differe…
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；State Key Lab of Intelligent Transportation System, Beihang University（北航智能交通系统国家重点实验室）
