DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation
Facebook（脸书）
Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jay… 展开作者
Mike Lambeta, Po-Wei Chou, Stephen Tian, Brian Yang, Benjamin Maloon, Victoria Rose Most, Dave Stroud, Raymond Santos, Ahmad Byagowi, Gregg Kammerer, Dinesh Jayaraman, Roberto Calandra
备注：
8 pages, published in the IEEE Robotics and Automation Letters (RA-L)
Despite decades of research, general purpose in-hand manipulation remains one of the unsolved challenges of robotics. One of the contributing factors that limit current robotic manipulation systems is the difficulty of precisely sensing contact forces -- sensing and reasoning about contact forces are crucial to accurately control interactions with the environment. As a step towards enabling better robotic manipulation, we introduce DIGIT, an inexpensive, compact, and high-resolution tactile sensor geared towards in-hand manipulation. DIGIT improves upon past vision-based tactile sensors by miniaturizing the form factor to be mountable on multi-fingered hands, and by providing several design improvements that result in an easier, more repeatable manufacturing process, and enhanced reliability. We demonstrate the capabilities of the DIGIT sensor by training deep neural network model-based controllers to manipulate glass marbles in-hand with a multi-finger robotic hand. To provide the robotic community access to reliable and low-cost tactile sensors, we open-source the DIGIT design at https://digit.ml/.
This paper develops closed-loop tactile controllers for dexterous robotic manipulation with a dual-palm robotic system. Tactile dexterity is an approach to dexterous manipulation that plans for robot/object interactions that render interpretable tactile information for control. We divide the role of tactile control into two goals: 1) control the contact state between the end-effector and the object (contact/no-contact, stick/slip) by regulating the stability of planned contact configurations and monitoring undesired slip events; and 2) control the object state by tactile-based tracking and ite…
The Role of Touch: Towards Optimal Tactile Sensing Distribution in Anthropomorphic Hands for Dexterous In-Hand Manipulation
VisLab, Institute for Systems；Soft Robotics Lab National University of Singapore, Singapore；BRAIR Lab, BioRobotics Institute Scuola Superiore Sant'Anna, Pisa, Italia
Multi-fingered hands could be used to achieve many dexterous manipulation tasks, similarly to humans, and tactile sensing could enhance the manipulation stability for a variety of objects. However, tactile sensors on multi-fingered hands have a variety of sizes and shapes. Convolutional neural networks (CNN) can be useful for processing tactile information, but the information from multi-fingered hands needs an arbitrary pre-processing, as CNNs require a rectangularly shaped input, which may lead to unstable results. Therefore, how to process such complex shaped tactile information and utilize…
Incorporating effective tactile sensing and mechanical compliance is key towards enabling robust and safe operation of robots in unknown, uncertain and cluttered environments. Towards realizing this goal, we present a lightweight, easy-to-build, highly compliant dense geometry sensor and end effector that comprises an inflated latex membrane with a depth sensor behind it. We present the motivations and the hardware design for this Soft-bubble and demonstrate its capabilities through example tasks including tactile-object classification, pose estimation and tracking, and nonprehensile object ma…
University of California, Berkeley（加利福尼亚大学伯克利分校）；Facebook AI Research（Facebook人工智能研究院）
Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dyn…
Robotic tactile sensing provides a method of recognizing objects and their properties where vision fails. Prior work on tactile perception in robotic manipulation has frequently focused on exploratory procedures (EPs). However, the also-human-inspired technique of in-hand-manipulation can glean rich data in a fraction of the time of EPs. We propose a simple 3-DOF robotic hand design, optimized for object rolling tasks via a variable-width palm and associated control system. This system dynamically adjusts the distance between the finger bases in response to object behavior. Compared to fixed f…
The University of Melbourne（墨尔本大学）；St. Vincent’s Hospital（圣文森特医院）
Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are …
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
University of California San Diego（加利福尼亚大学圣迭戈分校）；University of Southern California（南加利福尼亚大学）
Tactile sensors provide information that can be used to learn and execute manipulation tasks. Different tasks, however, might require different levels of sensory information; which in turn likely affect learning rates and performance. This paper evaluates the role of tactile information on autonomous learning of manipulation with a simulated 3-finger tendon-driven hand. We compare the ability of the same learning algorithm (Proximal Policy Optimization, PPO) to learn two manipulation tasks (rolling a ball about the horizontal axis with and without rotational stiffness) with three levels of tac…
In this paper we address the challenge of exploration in deep reinforcement learning for robotic manipulation tasks. In sparse goal settings, an agent does not receive any positive feedback until randomly achieving the goal, which becomes infeasible for longer control sequences. Inspired by touch-based exploration observed in children, we formulate an intrinsic reward based on the sum of forces between a robot's force sensors and manipulation objects that encourages physical interaction. Furthermore, we introduce contact-prioritized experience replay, a sampling scheme that prioritizes contact…
In this paper, we present a method to manipulate unknown objects in-hand using tactile sensing without relying on a known object model. In many cases, vision-only approaches may not be feasible; for example, due to occlusion in cluttered spaces. We address this limitation by introducing a method to reorient unknown objects using tactile sensing. It incrementally builds a probabilistic estimate of the object shape and pose during task-driven manipulation. Our approach uses Bayesian optimization to balance exploration of the global object shape with efficient task completion. To demonstrate the …
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Lenovo CTO Organization（联想首席技术官组织）；Nanyang Technological University（南洋理工大学）；TeleAl, China Telecom（中国电信天翼人工智能公司）
High resolution tactile sensors are often bulky and have shape profiles that make them awkward for use in manipulation. This becomes important when using such sensors as fingertips for dexterous multi-fingered hands, where boxy or planar fingertips limit the available set of smooth manipulation strategies. High resolution optical based sensors such as GelSight have until now been constrained to relatively flat geometries due to constraints on illumination geometry.Here, we show how to construct a rounded fingertip that utilizes a form of light piping for directional illumination. Our sensors c…
Collocated tactile sensing is a fundamental enabling technology for dexterous manipulation. However, deformable sensors introduce complex dynamics between the robot, grasped object, and environment that must be considered for fine manipulation. Here, we propose a method to learn soft tactile sensor membrane dynamics that accounts for sensor deformations caused by the physical interaction between the grasped object and environment. Our method combines the perceived 3D geometry of the membrane with proprioceptive reaction wrenches to predict future deformations conditioned on robot action. Grasp…
University of Liverpool（利物浦大学）；Fuzhou University（福州大学）
Sensing contacts throughout the fingers is an essential capability for a robot to perform manipulation tasks in cluttered environments. However, existing tactile sensors either only have a flat sensing surface or a compliant tip with a limited sensing area. In this paper, we propose a novel optical tactile sensor, the GelTip, that is shaped as a finger and can sense contacts on any location of its surface. The sensor captures high-resolution and color-invariant tactile images that can be exploited to extract detailed information about the end-effector's interactions against manipulated objects…
Unified Visual-Tactile-Action Modeling from Human Demonstrations for Dexterous Manipulation
Shanghai Jiao Tong University（上海交通大学）；State Key Laboratory of General Artificial Intelligence, BIGAI（北京通用人工智能研究院 通用人工智能全国重点实验室）；Sharpa Robotics；Beijing Institute of Technology（北京理工大学）
This work studies the problem of shape reconstruction and object localization using a vision-based tactile sensor, GelSlim. The main contributions are the recovery of local shapes from contact, an approach to reconstruct the tactile shape of objects from tactile imprints, and an accurate method for object localization of previously reconstructed objects. The algorithms can be applied to a large variety of 3D objects and provide accurate tactile feedback for in-hand manipulation. Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accu…
TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot
Southern University of Science and Technology（南方科技大学）；University of Hong Kong（香港大学）；Tencent Robotics X（腾讯 Robotics X）；University of Cambridge（剑桥大学）；Imperial College London（帝国理工学院）
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics（TARS机器人实验室）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；CASIA（中国科学院自动化研究所）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）
Object pose estimation methods allow finding locations of objects in unstructured environments. This is a highly desired skill for autonomous robot manipulation as robots need to estimate the precise poses of the objects in order to manipulate them. In this paper, we investigate the problems of tactile pose estimation and manipulation for category-level objects. Our proposed method uses a Bayes filter with a learned tactile observation model and a deterministic motion model. Later, we train policies using deep reinforcement learning where the agents use the belief estimation from the Bayes fil…
Tactile-Conditioned Diffusion Policy for Force-Aware Robotic Manipulation
Department of Computer Science, Technical University of Darmstadt（德累斯顿技术大学计算机科学系）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Technical University of Darmstadt（德累斯顿技术大学认知科学中心）；Hessian Center for Artificial Intelligence (Hessian.AI), Darmstadt（黑森人工智能中心（Hessian.AI））；Robotics Institute Germany (RIG)（德国机器人研究所）
TACTIC: Understanding Tactile Encoders and Conditioning for Contact-rich Robot Manipulation Policies
University of Technology Nuremberg（纽伦堡工业大学）；Ludwig-Maximilians University Munich（慕尼黑路德维希-马克西米利安大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Deggendorf Institute of Technology（德根多夫理工学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Tromso（特罗姆瑟大学）；DLR-German Aerospace Center（德国航空航天中心）
FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills
School of Automation, Southeast University（东南大学自动化学院）；Key Laboratory of Measurement and Control of CSE, Ministry of Education（东南大学复杂工程系统测量与控制教育部重点实验室）；Department of Engineering, King’s College London（伦敦国王学院工程系）
TAMEn: Tactile-Aware Manipulation Engine for Closed-Loop Data Collection in Contact-Rich Tasks
Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）；OpenDriveLab at The University of Hong Kong（香港大学OpenDriveLab）；Shanghai Jiao Tong University（上海交通大学）；East China University of Science and Technology（华东理工大学）
University of Lincoln（林肯大学）；Politecnico di Milano（米兰理工大学）
This paper introduces a novel approach to address the problem of Physical Robot Interaction (PRI) during robot pushing tasks. The approach uses a data-driven forward model based on tactile predictions to inform the controller about potential future movements of the object being pushed, such as a strawberry stem, using a robot tactile finger. The model is integrated into a Deep Functional Predictive Control (d-FPC) system to control the displacement of the stem on the tactile finger during pushes. Pushing an object with a robot finger along a desired trajectory in 3D is a highly nonlinear and c…
This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by uti…
