接触丰富型操作要求机器人不仅要协调移动位置，还要协调交互过程中接触与力的变化。然而，大多数机器人策略主要将动作表示为运动轨迹，将触觉信号作为输入或辅助预测目标，未在动作表示中显式编码接触动力学。我们提出ILA（Interaction Latents as Actions，交互潜变量作为动作），其使用可解码的交互潜变量，共同捕获运动、接触和力施加的变化，作为策略动作单元，为动作生成、评估和离线改进提供共享表示。我们通过视觉条件下的联合动作-触觉重构，结合时间条件隐式解码器来学习这些潜变量。将它们作为流匹配生成目标，支持对预训练的视觉-语言-动作模型（VLAs）和世界动作模型（WAMs）进行后训练，以及从头开始训练流匹配策略。共享解码器将生成的潜变量映射为可执行动作。对于离线改进，我们从专家演示编码的交互潜变量中学习触觉条件专家参考模型。所得的专家一致性奖励指导从成功 rollout 中进行加权学习，而与观测兼容的专家参考则指导纠正失败 rollout 中低奖励的预测。在三个策略族中，与触觉条件直接动作基线相比，ILA在四个真实世界任务上的平均成功率提高了21.0至36.5个百分点。消融实验强调了在动作目标中编码接触变化（超越触觉条件）的益处，同时相同的潜变量空间支持专家引导的离线改进。项目网站：this https URL。
英文摘要：
Contact-rich manipulation requires robots to coordinate not only where to move, but also how contact and force should evolve throughout an interaction. Yet most robot policies represent actions primarily as motion trajectories, using tactile signals as inputs or auxiliary prediction targets without explicitly encoding contact dynamics in the action representation. We propose ILA (Interaction Latents as Actions), which uses decodable interaction latents that jointly capture the evolution of motion, contact, and force application as policy action units, providing a shared representation for action generation, evaluation, and offline improvement. We learn these latents through visually conditioned joint action-tactile reconstruction with a time-conditioned implicit decoder. Using them as flow-matching generation targets supports post-training of pretrained vision-language-action models (VLAs) and world action models (WAMs), as well as training flow-matching policies from scratch. A shared decoder maps generated latents to executable actions. For offline improvement, we learn a tactile-conditioned expert reference model from interaction latents encoded from expert demonstrations. The resulting expert-consistency rewards guide weighted learning from successful rollouts, while observation-compatible expert references guide correction of low-reward predictions from failed rollouts. Across three policy families, ILA improves success rates averaged over four real-world tasks by 21.0-36.5 percentage points relative to tactile-conditioned direct-action baselines. Ablations highlight the benefit of encoding contact evolution in action targets beyond tactile conditioning, while the same latent space supports expert-guided offline improvement. Project website: https://mrsecant.github.io/ILA/.
TORL-VLA: Tactile Guided Online Reinforcement Learning for Contact-Rich Manipulation
Meituan（美团）；Beijing Institute of Technology（北京理工大学）；Beihang University（北京航空航天大学）；State Key Lab of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；China University of Mining and Technology (Beijing)（中国矿业大学（北京））
FWBC-VLA: Force-Aware Whole-Body Compensation for Contact-Rich Loco-Manipulation
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Deep Robotics（深度机器人公司）；Zhejiang University of Science and Technology（浙江科技大学）
This paper presents Contact Mode Guided Manipulation Planning (CMGMP) for 3D quasistatic and quasidynamic rigid body motion planning in dexterous manipulation. The CMGMP algorithm generates hybrid motion plans including both continuous state transitions and discrete contact mode switches, without the need for pre-specified contact sequences or pre-designed motion primitives. The key idea is to use automatically enumerated contact modes of environment-object contacts to guide the tree expansions during the search. Contact modes automatically synthesize manipulation primitives, while the samplin…
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
This paper uses a mobile manipulator with a collaborative robotic arm to manipulate objects beyond the robot's maximum payload. It proposes a single-shot probabilistic roadmap-based method to plan and optimize manipulation motion with environment support. The method uses an expanded object mesh model to examine contact and randomly explores object motion while keeping contact and securing affordable grasping force. It generates robotic motion trajectories after obtaining object motion using an optimization-based algorithm. With the proposed method's help, we can plan contact-rich manipulation …
Dexterous manipulation tasks often require contact switching, where fingers make and break contact with the object. We propose a method that plans trajectories for dexterous manipulation tasks involving contact switching using contact-implicit trajectory optimization (CITO) augmented with a high-level discrete contact sequence planner. We first use the high-level planner to find a sequence of finger contact switches given a desired object trajectory. With this contact sequence plan, we impose additional constraints in the CITO problem. We show that our method finds trajectories approximately 7…
When limbs are decoupled, we find that trajectory outcomes in mechanical systems subject to unilateral constraints vary differentiably with respect to initial conditions, even as the contact mode sequence varies.
Variable impedance control in operation-space is a promising approach to learning contact-rich manipulation behaviors. One of the main challenges with this approach is producing a manipulation behavior that ensures the safety of the arm and the environment. Such behavior is typically implemented via a reward function that penalizes unsafe actions (e.g. obstacle collision, joint limit extension), but that approach is not always effective and does not result in behaviors that can be reused in slightly different environments. We show how to combine Riemannian Motion Policies, a class of policies …
The discontinuities and multi-modality introduced by contacts make manipulation planning challenging. Many previous works avoid this problem by pre-designing a set of high-level motion primitives like grasping and pushing. However, such motion primitives are often not adequate to describe dexterous manipulation motions. In this work, we propose a method for dexterous manipulation planning at a more primitive level. The key idea is to use contact modes to guide the search in a sampling-based planning framework. Our method can automatically generate contact transitions and motion trajectories un…
Legged robots have great potential to perform loco-manipulation tasks, yet it is challenging to keep the robot balanced while it interacts with the environment. In this paper we study the use of additional contact points for maximising the robustness of loco-manipulation motions. Specifically, body-ground contact is studied for enhancing robustness and manipulation capabilities of quadrupedal robots. We propose to equip the robot with prongs: small legs rigidly attached to the body which ensure body-ground contact occurs in controllable point-contacts. The effect of these prongs on robustness …
Recent studies on quadruped robots have focused on either locomotion or mobile manipulation using a robotic arm. Legged robots can manipulate heavier and larger objects using non-prehensile manipulation primitives, such as planar pushing, to drive the object to the desired location. In this paper, we present a novel hierarchical model predictive control (MPC) for contact optimization of the manipulation task. Using two cascading MPCs, we split the loco-manipulation problem into two parts: the first to optimize both contact force and contact location between the robot and the object, and the se…
We present an approach to robotic manipulation of unknown objects through regulation of the object's contact configuration: the location, geometry, and mode of all contacts between the object, robot, and environment. A contact configuration constrains the forces and motions that can be applied to the object; however, synthesizing these constraints generally requires knowledge of the object's pose and geometry. We develop an object-agnostic approach for estimation and control that circumvents this need. Our framework directly estimates a set of wrench and motion constraints which it uses to reg…
Contact SLAM: An Active Tactile Exploration Policy Based on Physical Reasoning Utilized in Robotic Fine Blind Manipulation Tasks
Research Center for Intelligent Robotics（智能机器人研究中心）；National Key Laboratory of Aerospace Flight Dynamics（航空飞行动力学国家重点实验室）；Shaanxi Province Innovation Team of Intelligent Robotic Technology（陕西省智能机器人技术创新团队）；School of Astronautics, Northwestern Polytechnical University（航空学院，西北工业大学）
Should We Learn Contact-Rich Manipulation Policies from Sampling-Based Planners?
New York University（纽约大学）；Boston Dynamics AI Institute（波士顿动力AI研究院）；Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）；Artificial and Natural Intelligence Toulouse Institute（图卢兹人工智能与自然智能研究所）
Sunrise Setting Ltd（日出设置有限公司）；SAGE Publications Ltd（SAGE出版有限公司）；University of Birmingham（伯明翰大学）；Honda Research Institute Europe GmbH（本田欧洲研究院有限公司）
We describe a framework for changing-contact robot manipulation tasks that require the robot to make and break contacts with objects and surfaces. The discontinuous interaction dynamics of such tasks make it difficult to construct and use a single dynamics model or control strategy, and the highly non-linear nature of the dynamics during contact changes can be damaging to the robot and the objects. We present an adaptive control framework that enables the robot to incrementally learn to predict contact changes in a changing contact task, learn the interaction dynamics of the piece-wise continu…
In this paper, we analyze the effects of contact models on contact-implicit trajectory optimization for manipulation. We consider three different approaches: (1) a contact model that is based on complementarity constraints, (2) a smooth contact model, and our proposed method (3) a variable smooth contact model. We compare these models in simulation in terms of physical accuracy, quality of motions, and computation time. In each case, the optimization process is initialized by setting all torque variables to zero, namely, without a meaningful initial guess. For simulations, we consider a pushin…
We present a novel method for the direct transfer of grasps and manipulations between objects and hands through utilization of contact areas. Our method fully preserves contact shapes, and in contrast to existing techniques, is not dependent on grasp families, requires no model training or grasp sampling, makes no assumptions about manipulator morphology or kinematics, and allows user control over both transfer parameters and solution optimization. Despite these accommodations, we show that our method is capable of synthesizing kinematically feasible whole hand poses in seconds even for poor i…
Robotics and Biology Laboratory, Technische Universität Berlin（柏林工业大学机器人与生物学实验室）
This paper evaluates state-of-the-art contact models at predicting the motions and forces involved in simple in-hand robotic manipulations. In particular it focuses on three primitive actions --linear sliding, pivoting, and rolling-- that involve contacts between a gripper, a rigid object, and their environment. The evaluation is done through thousands of controlled experiments designed to capture the motion of object and gripper, and all contact forces and torques at 250Hz. We demonstrate that a contact modeling approach based on Coulomb's friction law and maximum energy principle is effectiv…
The empirical success of Reinforcement Learning (RL) in the setting of contact-rich manipulation leaves much to be understood from a model-based perspective, where the key difficulties are often attributed to (i) the explosion of contact modes, (ii) stiff, non-smooth contact dynamics and the resulting exploding / discontinuous gradients, and (iii) the non-convexity of the planning problem. The stochastic nature of RL addresses (i) and (ii) by effectively sampling and averaging the contact modes. On the other hand, model-based methods have tackled the same challenges by smoothing contact dynami…
Learning When to See and When to Feel: Adaptive Vision-Torque Fusion for Contact-Aware Manipulation
Edwardson School of Industrial Engineering, Purdue University（普渡大学爱德华森工业工程学院）；Zachry Department of Civil and Environmental Engineering, Texas A&M University（德州农工大学扎克里土木与环境工程系）
ETH Zurich（苏黎世联邦理工学院）；Robotic Systems Lab（机器人系统实验室）
We present a reformulation of a contact-implicit optimization (CIO) approach that computes optimal trajectories for rigid-body systems in contact-rich settings. A hard-contact model is assumed, and the unilateral constraints are imposed in the form of complementarity conditions. Newton's impact law is adopted for enhanced physical correctness. The optimal control problem is formulated as a multi-staged program through a multiple-shooting scheme. This problem structure is exploited within the FORCES Pro framework to retrieve optimal motion plans, contact sequences and control inputs with increa…
Worcester Polytechnic Institute（伍斯特理工学院）；Imperial College London（伦敦帝国理工学院）
Attempts to achieve robotic Within-Hand-Manipulation (WIHM) generally utilize either high-DOF robotic hands with elaborate sensing apparatus or multi-arm robotic systems. In prior work we presented a simple robot hand with variable friction robot fingers, which allow a low-complexity approach to within-hand object translation and rotation, though this manipulation was limited to planar actions. In this work we extend the capabilities of this system to 3D manipulation with a novel region-based WIHM planning algorithm and utilizing extrinsic contacts. The ability to modulate finger friction enha…
Massachusetts Institute of Technology（麻省理工学院）；Department of Mechanical Engineering（机械工程系（不提取，因为系属于细拆））
This paper presents an approach to in-hand manipulation planning that exploits the mechanics of alternating sticking contact. Particularly, we consider the problem of manipulating a grasped object using external pushes for which the pusher sticks to the object. Given the physical properties of the object, frictional coefficients at contacts and a desired regrasp on the object, we propose a sampling-based planning framework that builds a pushing strategy concatenating different feasible stable pushes to achieve the desired regrasp. An efficient dynamics formulation allows us to plan in-hand man…
AeRSoM: An Aerial Rigid-Soft Integrated Manipulator for Contact-Rich Manipulation
Hunan University（湖南大学）；National Engineering Research Center for Robot Visual Perception and Control Technology（国家机器人视觉感知与控制技术工程研究中心）；Beihang University（北京航空航天大学）
Carnegie Mellon University（卡内基梅隆大学）；Southern University of Science and Technology（南方科技大学）
A shared grasp is a grasp formed by contacts between the manipulated object and both the robot hand and the environment. By trading off hand contacts for environmental contacts, a shared grasp requires fewer contacts with the hand, and enables manipulation even when a full grasp is not possible. Previous research has used shared grasps for non-prehensile manipulation such as pivoting and tumbling. This paper treats the problem more generally, with methods to select the best shared grasp and robot actions for a desired object motion. The central issue is to evaluate the feasible contact modes: …
KTH Royal Institute of Technology（瑞典皇家理工学院）；Chalmers University of Technology（查尔姆斯理工大学）
Deep Neural Networks (NNs) have been widely utilized in contact-rich manipulation tasks to model the complicated contact dynamics. However, NN-based models are often difficult to decipher which can lead to seemingly inexplicable behaviors and unidentifiable failure cases. In this work, we address the interpretability of NN-based models by introducing the kinodynamic images. We propose a methodology that creates images from the kinematic and dynamic data of a contact-rich manipulation task. Our formulation visually reflects the task's state by encoding its kinodynamic variations and temporal ev…
