Whole-body humanoid manipulation of bulky, deformable, and shared-load objects requires distributed contact sensing and explicit force regulation, yet most imitation policies treat contact force only implicitly. On the other hand, different demonstration sources provide complementary modalities with inherent trade-offs: human demonstrations capture natural contact forces but not robot-executable actions, while teleoperation directly records robot actions but with less natural force regulation. This paper presents \textbf{WT-UMI}, a wearable whole-body tactile interface worn by human operators or mounted on humanoids, providing accurate observations of tactile images, contact forces, and end-effector poses across both human demonstration and humanoid teleoperation modes. We introduce a force-conditioned target-pose correction module that converts measured human poses into contact-aware robot targets by learning corrections from teleoperation data. To leverage the natural force interaction in human data, we propose a force-supervised planner that predicts end-effector pose chunks and contact-force trajectories. The predicted contact force serves as the reference for a tactile-based admittance controller. Across five contact-rich tasks spanning deformable objects, bulky rigid objects, and human--humanoid collaboration, WT-UMI improves success rate and reduces contact-position tracking error over four policy baselines. Our project page is available at https://wt-umi.github.io/WTUMI/.
Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Technical University of Darmstadt（德累斯顿技术大学认知科学中心）；Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt（黑森人工智能中心（Hessian.AI））；Robotics Institute Germany (RIG)（德国机器人研究所）
Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize…
Incorporating effective tactile sensing and mechanical compliance is key towards enabling robust and safe operation of robots in unknown, uncertain and cluttered environments. Towards realizing this goal, we present a lightweight, easy-to-build, highly compliant dense geometry sensor and end effector that comprises an inflated latex membrane with a depth sensor behind it. We present the motivations and the hardware design for this Soft-bubble and demonstrate its capabilities through example tasks including tactile-object classification, pose estimation and tracking, and nonprehensile object ma…
The University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are …
Tencent Robotics X（腾讯机器人X实验室）；Imperial College London（帝国理工学院）
Reinforcement Learning (RL) methods have been widely applied for robotic manipulations via sim-to-real transfer, typically with proprioceptive and visual information. However, the incorporation of tactile sensing into RL for contact-rich tasks lacks investigation. In this paper, we model a tactile sensor in simulation and study the effects of its feedback in RL-based robotic control via a zero-shot sim-to-real approach with domain randomization. We demonstrate that learning and controlling with feedback from tactile sensor arrays at the gripper, both in simulation and reality, can enhance gras…
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
Towards Accurate End-Effector Localization for UMI-Style Robotic Manipulation Teaching
Chongqing University（重庆大学）；Northwestern Polytechnical University（西北工业大学）；The University of Tokyo（东京大学）；Jiaxing University（嘉兴大学）；Sichuan University（四川大学）；Beijing Institute of Technology（北京理工大学）；Shanghai Jiao Tong University（上海交通大学）
Compliant In-hand Rolling Manipulation Using Tactile Sensing
Center for Robotics and Biosystems and the Department of Mechanical Engineering, Northwestern University（机器人与生物系统中心和机械工程系，西北大学）；Northwestern Institute on Complex Systems (NICO)（复杂系统研究所（NICO））
TacForeSight: Force-Guided Tactile World Model for Contact-Rich Manipulation
TARS Robotics；National University of Singapore（新加坡国立大学）；Shanghai Jiao Tong University（上海交通大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Fudan University（复旦大学）
Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation
Interactive Robot Perception & Learning, TU Darmstadt（互动机器人感知与学习，图腾达姆施塔特大学）；Intelligent Autonomous Systems, TU Darmstadt（智能自主系统，图腾达姆施塔特大学）；Hessian AI（黑森人工智能）；Robotics Institute Germany（德国机器人研究所）
In this paper, we present a method to manipulate unknown objects in-hand using tactile sensing without relying on a known object model. In many cases, vision-only approaches may not be feasible; for example, due to occlusion in cluttered spaces. We address this limitation by introducing a method to reorient unknown objects using tactile sensing. It incrementally builds a probabilistic estimate of the object shape and pose during task-driven manipulation. Our approach uses Bayesian optimization to balance exploration of the global object shape with efficient task completion. To demonstrate the …
Robot control for tactile feedback-based manipulation can be difficult due to the modeling of physical contacts, partial observability of the environment, and noise in perception and control. This work focuses on solving partial observability of contact-rich manipulation tasks as a Sequence-to-Sequence (Seq2Seq)} Imitation Learning (IL) problem. The proposed Seq2Seq model produces a robot-environment interaction sequence to estimate the partially observable environment state variables. Then, the observed interaction sequence is transformed to a control sequence for the task itself. The propose…
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Lenovo CTO Organization（联想首席技术官组织）；Nanyang Technological University（南洋理工大学）；TeleAl, China Telecom（中国电信天翼人工智能公司）
High resolution tactile sensors are often bulky and have shape profiles that make them awkward for use in manipulation. This becomes important when using such sensors as fingertips for dexterous multi-fingered hands, where boxy or planar fingertips limit the available set of smooth manipulation strategies. High resolution optical based sensors such as GelSight have until now been constrained to relatively flat geometries due to constraints on illumination geometry.Here, we show how to construct a rounded fingertip that utilizes a form of light piping for directional illumination. Our sensors c…
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
University of Liverpool（利物浦大学）；Fuzhou University（福州大学）
Sensing contacts throughout the fingers is an essential capability for a robot to perform manipulation tasks in cluttered environments. However, existing tactile sensors either only have a flat sensing surface or a compliant tip with a limited sensing area. In this paper, we propose a novel optical tactile sensor, the GelTip, that is shaped as a finger and can sense contacts on any location of its surface. The sensor captures high-resolution and color-invariant tactile images that can be exploited to extract detailed information about the end-effector's interactions against manipulated objects…
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University（上海交通大学ScaleLab）；D-Robotics；ViTai Robotics；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
VT-WAM: Visual-Tactile World Action Model for Contact-Rich Manipulation
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所复杂系统管理与控制国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；TARS Robotics；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）
