Deformable object manipulation requires computationally efficient representations that are compatible with robotic sensing modalities. In this paper, we present VIRDO:an implicit, multi-modal, and continuous representation for deformable-elastic objects. VIRDO operates directly on visual (point cloud) and tactile (reaction forces) modalities and learns rich latent embeddings of contact locations and forces to predict object deformations subject to external contacts.Here, we demonstrate VIRDOs ability to: i) produce high-fidelity cross-modal reconstructions with dense unsupervised correspondences, ii) generalize to unseen contact formations,and iii) state-estimation with partial visio-tactile feedback
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables…
AetheRock: An Arm-Worn Robot Teaching System for Force-Guided Vision-Tactile Learning
Shanghai Jiao Tong University（上海交通大学）；Ant Group（蚂蚁集团）；Shanghai Innovation Institute（上海创新研究院）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）
UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Models
Harbin Institute of Technology（哈尔滨工业大学）；Great Bay University（大湾区大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；Nanjing University（南京大学）；Daimon Robotics
École de technologie supérieure（高等技术学院）；Bordeaux Institute of Technology（波尔多理工学院）；ENSEIRB-MATMECA（波尔多国立高等电子、信息与无线电通信学院-波尔多国立高等机械与航空技术学院）
Combining 3D vision with tactile sensing could unlock a greater level of dexterity for robots and improve several manipulation tasks. However, obtaining a close-up 3D view of the location where manipulation contacts occur can be challenging, particularly in confined spaces, cluttered environments, or without installing more sensors on the end effector. In this context, this paper presents StereoTac, a novel vision-based sensor that combines tactile sensing with 3D vision. The proposed sensor relies on stereoscopic vision to capture a 3D representation of the environment before contact and uses…
The importance of force perception in interacting with the environment was proven years ago. However, it is still a challenge to measure the contact force distribution accurately in real-time. In order to break through this predicament, we propose a new vision-based tactile sensor, the Tac3D sensor, for measuring the three-dimensional contact surface shape and contact force distribution. In this work, virtual binocular vision is first applied to the tactile sensor, which allows the Tac3D sensor to measure the three-dimensional tactile information in a simple and efficient way and has the advan…
The sense of touch plays a key role in enabling humans to understand and interact with surrounding environments. For robots, tactile sensing is also irreplaceable. While interacting with objects, tactile sensing provides useful information for the robot to understand the object, such as distributed pressure, temperature, vibrations and texture. During robot grasping, vision is often occluded by its end-effectors, whereas tactile sensing can measure areas that are not accessible by vision. In the past decades, a number of tactile sensors have been developed for robots and used for different rob…
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（香港科技大学深港协同创新研究院）
Tactile sensing is an essential perception for robots to complete dexterous tasks. As a promising tactile sensing technique, vision-based tactile sensors have been developed to improve robot performance in manipulation and grasping. Here we propose a new design of a vision-based tactile sensor, DelTact. The sensor uses a modular hardware architecture for compactness whilst maintaining a contact measurement of full resolution (798*586) and large area (675mm2). Moreover, it adopts an improved dense random color pattern based on the previous version to achieve high accuracy of contact deformation…
Shanghai Qi Zhi Institute（上海期智研究院）；Tsinghua University（清华大学）；Massachusetts Institute of Technology（麻省理工学院）
Vision-based tactile sensors that can measure 3D geometry of the contacting objects are crucial for robots to perform dexterous manipulation tasks. However, the existing sensors are usually complicated to fabricate and delicate to extend. In this work, we novelly take advantage of the reflection property of semitransparent elastomer to design a robust, low-cost, and easy-to-fabricate tactile sensor named DTact. DTact measures high-resolution 3D geometry accurately from the darkness shown in the captured tactile images with only a single image for calibration. In contrast to previous sensors, D…
DTactive: A Vision-Based Tactile Sensor with Active Surface
Shanghai Qi Zhi Institute（上海期智研究院）；Huazhong University of Science and Technology（华中科技大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
MVP-Tac: A Miniaturized Dual-Modal Vision and Photoelastic Tactile Sensor for Robot-Assisted Minimally Invasive Surgery
Elmore Family School of Electrical and Computer Engineering, Purdue University（普渡大学埃尔莫尔电气与计算机工程学院）；Edwardson School of Industrial Engineering, Purdue University（普渡大学爱德华森工业工程学院）；School of Mechanical Engineering, Purdue University（普渡大学机械工程学院）；Robotics Engineering Technology, Purdue University（普渡大学机器人工程技术学院）；Department of Civil Engineering, Stony Brook University（石溪大学土木工程系）；Department of Mechanical Engineering, Michigan State University（密歇根州立大学机械工程系）
University of Liverpool（利物浦大学）；King’s College London（伦敦国王学院）；Monash University（莫纳什大学）
Picking up transparent objects is still a challenging task for robots. The visual properties of transparent objects such as reflection and refraction make the current grasping methods that rely on camera sensing fail to detect and localise them. However, humans can handle the transparent object well by first observing its coarse profile and then poking an area of interest to get a fine profile for grasping. Inspired by this, we propose a novel framework of vision-guided tactile poking for transparent objects grasping. In the proposed framework, a segmentation network is first used to predict t…
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑工业大学机器人与系统智能教席，慕尼黑机器人与机器智能研究所）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）
Coordinating proximity and tactile imaging by collocating cameras with tactile sensors can 1) provide useful information before contact such as object pose estimates and visually servo a robot to a target with reduced occlusion and higher resolution compared to head-mounted or external depth cameras, 2) simplify the contact point and pose estimation problems and help tactile sensing avoid erroneous matches when a surface does not have significant texture or has repetitive texture with many possible matches, and 3) use tactile imaging to further refine contact point and object pose estimation. …
Johns Hopkins School of Medicine（约翰斯·霍普金斯大学医学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）
The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, able-bodied individuals inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we built two fabric-based tactile sensors that measure contact location along the palmar and…
Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
College of Big Data and Internet, Shenzhen Technology University, China（大数据与互联网学院，深圳科技大学，中国）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, China（中德智能制造学院，深圳科技大学，中国）；Robotics Department, Institute of Smart Systems and Artificial Intelligence (ISSAI), Nazarbayev University, Kazakhstan（机器人系，智能系统与人工智能研究所（ISSAI），纳扎尔巴耶夫大学，哈萨克斯坦）；Foundation Robotics Labs, Germany（基础机器人实验室，德国）
Hong Kong University of Science and Technology（香港科技大学）
Retrieving rich contact information from robotic tactile sensing has been a challenging, yet significant task for the effective perception of object properties that the robot interacts with. This work is dedicated to developing an algorithm to estimate contact force and torque for vision-based tactile sensors. We first introduce the observation of the contact deformation patterns of hyperelastic materials under ideal single-axial loads in simulation. Then based on the observation, we propose a method of estimating surface forces and torque from the contact deformation vector field with the Hel…
Carnegie Mellon University（卡内基梅隆大学）；Brigham Young University（杨百翰大学）
Knowledge of 3-D object shape is of great importance to robot manipulation tasks, but may not be readily available in unstructured environments. While vision is often occluded during robot-object interaction, high-resolution tactile sensors can give a dense local perspective of the object. However, tactile sensors have limited sensing area and the shape representation must faithfully approximate non-contact areas. In addition, a key challenge is efficiently incorporating these dense tactile measurements into a 3-D mapping framework. In this work, we propose an incremental shape mapping method …
Vision-Based Tactile Intelligence for Robotics: Sensing, Learning, and Embodied Manipulation
Great Bay University（大湾区大学）；Tsinghua University（清华大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong Polytechnic University（香港理工大学）；South China University of Technology（华南理工大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；King’s College London（伦敦国王学院）
Representation-Aligned Tactile Grounding for Contact-Rich Robotic Manipulation
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院智能信息处理上海市重点实验室）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；Lenovo CTO Organization（联想首席技术官组织）；Nanyang Technological University（南洋理工大学）；TeleAl, China Telecom（中国电信天翼人工智能公司）
TacVLA: Contact-Aware Tactile Fusion for Robust Vision-Language-Action Manipulation
Human-Robot Interfaces and Interaction Lab, Istituto Italiano di Tecnologia（人机交互实验室，意大利理工学院）；Ph.D. program of national interest in Robotics and Intelligent Machines (DRIM) and Università di Genova（机器人与智能机器国家级博士项目（DRIM）和热那亚大学）；Edwardson School of Industrial Engineering, Purdue University（工业工程学院，普渡大学）
Hong Kong University of Science and Technology（香港科技大学）；Nanjing University of Information Science and Technology（南京信息工程大学）
Monitoring the state of contact is essential for robotic devices, especially grippers that implement gecko-inspired adhesives where intimate contact is crucial for a firm attachment. However, due to the lack of deformable sensors, few have demonstrated tactile sensing for gecko grippers. We present Viko, an adaptive gecko gripper that utilizes vision-based tactile sensors to monitor contact state. The sensor provides high-resolution real-time measurements of contact area and shear force. Moreover, the sensor is adaptive, low-cost, and compact. We integrated gecko-inspired adhesives into the se…
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；The Imperial College of Science, Technology and Medicine（帝国理工学院）；Donders Institute for Brain and Cognition, Radboud University（拉德堡德大学唐德斯脑认知研究所）
This work presents a novel active visuo-tactile based framework for robotic systems to accurately estimate pose of objects in dense cluttered environments. The scene representation is derived using a novel declutter graph (DG) which describes the relationship among objects in the scene for decluttering by leveraging semantic segmentation and grasp affordances networks. The graph formulation allows robots to efficiently declutter the workspace by autonomously selecting the next best object to remove and the optimal action (prehensile or non-prehensile) to perform. Furthermore, we propose a nove…
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；Samsung R&D Institute China–Beijing（三星中国北京研发中心）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
We investigate how high-resolution tactile sensors can be utilized in combination with vision and depth sensing, to improve grasp stability prediction. Recent advances in simulating high-resolution tactile sensing, in particular the TACTO simulator, enabled us to evaluate how neural networks can be trained with a combination of sensing modalities. With the large amounts of data needed to train large neural networks, robotic simulators provide a fast way to automate the data collection process. We expand on the existing work through an ablation study and an increased set of objects taken from t…
Nagoya University（名古屋大学）；Technical University of Darmstadt（达姆施塔特工业大学）
Vision-based tactile sensors have gained extensive attention in the robotics community. The sensors are highly expected to be capable of extracting contact information i.e. haptic information during in-hand manipulation. This nature of tactile sensors makes them a perfect match for haptic feedback applications. In this paper, we propose a contact force estimation method using the vision-based tactile sensor DIGIT, and apply it to a position-force teleoperation architecture for force feedback. The force estimation is done by building a depth map for DIGIT gel surface deformation measurement and…
9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation
Shanghai Qi Zhi Institute（上海期智研究院）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
