Tactile sensors provide information that can be used to learn and execute manipulation tasks. Different tasks, however, might require different levels of sensory information; which in turn likely affect learning rates and performance. This paper evaluates the role of tactile information on autonomous learning of manipulation with a simulated 3-finger tendon-driven hand. We compare the ability of the same learning algorithm (Proximal Policy Optimization, PPO) to learn two manipulation tasks (rolling a ball about the horizontal axis with and without rotational stiffness) with three levels of tactile sensing: no sensing, 1D normal force, and 3D force vector. Surprisingly, and contrary to recent work on manipulation, adding 1D force-sensing did not always improve learning rates compared to no sensing---likely due to whether or not normal force is relevant to the task. Nonetheless, even though 3D force-sensing increases the dimensionality of the sensory input---which would in general hamper algorithm convergence---it resulted in faster learning rates and better performance. We conclude that, in general, sensory input is useful to learning only when it is relevant to the task---as is the case of 3D force-sensing for in-hand manipulation against gravity. Moreover, the utility of 3D force-sensing can even offset the added computational cost of learning with higher-dimensional sensory input.
Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize…
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
In this paper we address the challenge of exploration in deep reinforcement learning for robotic manipulation tasks. In sparse goal settings, an agent does not receive any positive feedback until randomly achieving the goal, which becomes infeasible for longer control sequences. Inspired by touch-based exploration observed in children, we formulate an intrinsic reward based on the sum of forces between a robot's force sensors and manipulation objects that encourages physical interaction. Furthermore, we introduce contact-prioritized experience replay, a sampling scheme that prioritizes contact…
Evetac: An Event-based Optical Tactile Sensor for Robotic Manipulation
Hessian Centre for Artificial Intelligence（黑森人工智能中心）；Technische Universität Dresden（德累斯顿工业大学）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）；Technical University of Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, TU Darmstadt（达姆施塔特工业大学认知科学中心）
In this paper, we present a method to manipulate unknown objects in-hand using tactile sensing without relying on a known object model. In many cases, vision-only approaches may not be feasible; for example, due to occlusion in cluttered spaces. We address this limitation by introducing a method to reorient unknown objects using tactile sensing. It incrementally builds a probabilistic estimate of the object shape and pose during task-driven manipulation. Our approach uses Bayesian optimization to balance exploration of the global object shape with efficient task completion. To demonstrate the …
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Lenovo CTO Organization（联想首席技术官组织）；Nanyang Technological University（南洋理工大学）；TeleAl, China Telecom（中国电信天翼人工智能公司）
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
University of Liverpool（利物浦大学）；Fuzhou University（福州大学）
Sensing contacts throughout the fingers is an essential capability for a robot to perform manipulation tasks in cluttered environments. However, existing tactile sensors either only have a flat sensing surface or a compliant tip with a limited sensing area. In this paper, we propose a novel optical tactile sensor, the GelTip, that is shaped as a finger and can sense contacts on any location of its surface. The sensor captures high-resolution and color-invariant tactile images that can be exploited to extract detailed information about the end-effector's interactions against manipulated objects…
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology（南方科技大学）；University of Hong Kong（香港大学）；Tencent Robotics X（腾讯 Robotics X）；University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics（TARS机器人实验室）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；CASIA（中国科学院自动化研究所）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）
Object pose estimation methods allow finding locations of objects in unstructured environments. This is a highly desired skill for autonomous robot manipulation as robots need to estimate the precise poses of the objects in order to manipulate them. In this paper, we investigate the problems of tactile pose estimation and manipulation for category-level objects. Our proposed method uses a Bayes filter with a learned tactile observation model and a deterministic motion model. Later, we train policies using deep reinforcement learning where the agents use the belief estimation from the Bayes fil…
Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Technical University of Darmstadt（德累斯顿技术大学认知科学中心）；Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt（黑森人工智能中心（Hessian.AI））；Robotics Institute Germany (RIG)（德国机器人研究所）
TACTIC: Understanding Tactile Encoders and Conditioning for Contact-rich Robot Manipulation Policies
University of Technology Nuremberg（纽伦堡工业大学）；Ludwig-Maximilians University Munich（慕尼黑路德维希-马克西米利安大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Deggendorf Institute of Technology（德根多夫理工学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Tromso（特罗姆瑟大学）；DLR-German Aerospace Center（德国航空航天中心）
The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation
VisLab, Institute for Systems；Soft Robotics Lab National University of Singapore, Singapore；BRAIR Lab, BioRobotics Institute Scuola Superiore Sant'Anna, Pisa, Italia
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics. One of the contributing factors that limit current robotic manipulation systems is the difficulty of precisely sensing contact forces -- sensing and reasoning about contact forces are crucial to accurately control interactions with the environment. As a step towards enabling better robotic manipulation, we introduce DIGIT, an inexpensive, compact, and high-resolution tactile sensor geared towards in-hand manipulation. DIGIT improves upon past vision-based tactile sensors by min…
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University（东南大学自动化学院）；Key Laboratory of Measurement and Control of CSE, Ministry of Education（东南大学复杂工程系统测量与控制教育部重点实验室）；Department of Engineering, King’s College London（伦敦国王学院工程系）
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；OpenDriveLab at The University of Hong Kong（香港大学OpenDriveLab）；Shanghai Jiao Tong University（上海交通大学）；East China University of Science and Technology（华东理工大学）
Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation
Shanghai Jiao Tong University（上海交通大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院 通用人工智能全国重点实验室）；Sharpa Robotics；Beijing Institute of Technology（北京理工大学）
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology（智能制造装备与技术国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）；Wuhan Huaweike Intelligent Technology Co., Ltd.（武汉华为凯科技有限公司）
Tactile sensing is an essential capability for robots that carry out dexterous manipulation tasks. While cameras, Lidars and other remote sensors can assess a scene globally and instantly, tactile sensors can reduce their measurement uncertainties and gain information about the local physical interactions between the in-contact objects and the robot, that are often not accessible via remote sensing. Tactile sensors can be grouped into two main categories: electronic tactile skins and camera based optical tactile sensors. The former are slim and can be fitted to different body parts, whereas th…
