Active Visuo-Tactile Interactive Robotic Perception for Accurate Object Pose Estimation in Dense Clutter
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；The Imperial College of Science, Technology and Medicine（帝国理工学院）；Donders Institute for Brain and Cognition, Radboud University（拉德堡德大学唐德斯脑认知研究所）
This work presents a novel active visuo-tactile based framework for robotic systems to accurately estimate pose of objects in dense cluttered environments. The scene representation is derived using a novel declutter graph (DG) which describes the relationship among objects in the scene for decluttering by leveraging semantic segmentation and grasp affordances networks. The graph formulation allows robots to efficiently declutter the workspace by autonomously selecting the next best object to remove and the optimal action (prehensile or non-prehensile) to perform. Furthermore, we propose a novel translation-invariant Quaternion filter (TIQF) for active vision and active tactile based pose estimation. Both active visual and active tactile points are selected by maximizing the expected information gain. We evaluate our proposed framework on a system with two robots coordinating on randomized scenes of dense cluttered objects and perform ablation studies with static vision and active vision based estimation prior and post decluttering as baselines. Our proposed active visuo-tactile interactive perception framework shows upto 36% improvement in pose accuracy compared to the active vision baseline.
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University（上海交通大学ScaleLab）；D-Robotics；ViTai Robotics；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables…
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）
Humans can accurately determine whether the object in hand has slipped or not by visual and tactile perception. However, it is still a challenge for robots to detect in-hand object slip through visuo-tactile fusion. To address this issue, a novel visuo-tactile fusion deep neural network is proposed to detect slip, which is a time-dependent continuous action. By using the multi-scale temporal convolution network (MS-TCN) to extract the temporal features of visual and tactile data, the slip can be detected effectively. In this paper, a 7-dregree-of-freedom (7-DoF) robot manipulator equipped with…
In this work, we propose a novel, bio-inspired multi-sensory SLAM approach called ViTa-SLAM. Compared to other multisensory SLAM variants, this approach allows for a seamless multi-sensory information fusion whilst naturally interacting with the environment. The algorithm is empirically evaluated in a simulated setting using a biomimetic robot platform called the WhiskEye. Our results show promising performance enhancements over existing bio-inspired SLAM approaches in terms of loop-closure detection.
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics（TARS机器人实验室）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；CASIA（中国科学院自动化研究所）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；School of Information, Renmin University of China（中国人民大学信息学院）；Fudan University（复旦大学）；TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
Shanghai Innovation Institute（上海创新研究院）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；Shanghai University（上海大学）
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
UC Berkeley（加州大学伯克利分校）；TU Dresden（德累斯顿工业大学）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在环触觉互联网中心（CeTI））；Carnegie Mellon University（卡内基梅隆大学）
Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help us decide which actions we should choose and how to perform them. E.g., we can drive slower if we see that we have bad traction or grasp tighter if an item looks slippery. We believe that this ability also helps robots to enhance their understanding of the environment, and thus enables them to tailor their actions to the situation at hand. We therefore propose a model to estimate the degree of tactile properties from visual pe…
Many manipulation tasks require the robot to control the contact between a grasped compliant tool and the environment, e.g. scraping a frying pan with a spatula. However, modeling tool-environment interaction is difficult, especially when the tool is compliant, and the robot cannot be expected to have the full geometry and physical properties (e.g., mass, stiffness, and friction) of all the tools it must use. We propose a framework that learns to predict the effects of a robot's actions on the contact between the tool and the environment given visuo-tactile perception. Key to our framework is …
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑工业大学机器人与系统智能教席，慕尼黑机器人与机器智能研究所）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机科学学院，北京大学）；PKU-Agibot Lab, School of Computer Science, Peking University（北京大学计算机科学学院PKU-Agibot实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（国家多媒体信息处理重点实验室，计算机科学学院，北京大学）
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；Technische Universität München（慕尼黑工业大学）；Donders Institute for Brain and Cognition（唐德斯脑与认知研究所）；Radboud University（拉德堡德大学）
This paper proposes a novel active visuo-tactile based methodology wherein the accurate estimation of the time-invariant SE(3) pose of objects is considered for autonomous robotic manipulators. The robot equipped with tactile sensors on the gripper is guided by a vision estimate to actively explore and localize the objects in the unknown workspace. The robot is capable of reasoning over multiple potential actions, and execute the action to maximize information gain to update the current belief of the object. We formulate the pose estimation process as a linear translation invariant quaternion …
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（工程学院，坎帕尼亚路易吉·范维蒂利大学）；Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova（信息工程学院，帕多瓦大学）
Aalto University（阿尔托大学）；University of Bristol（布里斯托大学）；University of the West of England（西英格兰大学）
RatSLAM is a rat hippocampus-inspired visual Simultaneous Localization and Mapping (SLAM) framework capable of generating semi-metric topological representations of indoor and outdoor environments. Whisker-RatSLAM is a 6D extension of the RatSLAM and primarily focuses on object recognition by generating point clouds of objects based on whisking information. This paper introduces a novel extension to both former works that is referred to as ViTa-SLAM that harnesses both vision and tactile information for performing SLAM. This not only allows the robot to perform natural interaction with the env…
Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \cite{dosovitskiy2021image} to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus at…
Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
Dept. of Electronic Engineering at Sogang University（ソガン大学电子工程系）；Dept. of Immersive Media and Engineering at Sungkyunkwan University（顺天大学沉浸媒体与工程系）；College of Medicine, Yonsei University（延世大学医学院）
École de technologie supérieure（高等技术学院）；Bordeaux Institute of Technology（波尔多理工学院）；ENSEIRB-MATMECA（波尔多国立高等电子、信息与无线电通信学院-波尔多国立高等机械与航空技术学院）
Combining 3D vision with tactile sensing could unlock a greater level of dexterity for robots and improve several manipulation tasks. However, obtaining a close-up 3D view of the location where manipulation contacts occur can be challenging, particularly in confined spaces, cluttered environments, or without installing more sensors on the end effector. In this context, this paper presents StereoTac, a novel vision-based sensor that combines tactile sensing with 3D vision. The proposed sensor relies on stereoscopic vision to capture a 3D representation of the environment before contact and uses…
DexViTac: Collecting Human Visuo-Tactile-Kinematic Demonstrations for Contact-Rich Dexterous Manipulation
State Key Laboratory of Intelligent Manufacturing Equipment and Technology（智能制造装备与技术国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）；Wuhan Huaweike Intelligent Technology Co., Ltd.（武汉华为凯科技有限公司）
UC Berkeley（加州大学伯克利分校）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Autodesk（欧特克）；Autodesk Research（欧特克研究院）
Industrial insertion tasks are often performed repetitively with parts that are subject to tight tolerances and prone to breakage. Learning an industrial insertion policy in real is challenging as the collision between the parts and the environment can cause slippage or breakage of the part. In this paper, we present a safe self-supervised method to learn a visuo-tactile insertion policy that is robust to grasp pose variations. The method reduces human input and collisions between the part and the receptacle. The method divides the insertion task into two phases. In the first align phase, a ta…
SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
Tuojing Intelligence（拓境智能）；Tsinghua University（清华大学）；Southeast University（东南大学）；Stevens Institute of Technology（斯蒂文斯理工学院）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Manchester（曼彻斯特大学）；Simple AI；Imperial College London（帝国理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）；The University of Hong Kong（香港大学）
Georgia Institute of Technology（佐治亚理工学院）；NVIDIA（英伟达）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model track…
CAAT: Contact-Aware Attention Scaling and Tactile Masking for Data-Efficient Contact-Rich Manipulation
ShanghaiTech University（上海科技大学）；Beihang University（北京航空航天大学）；Beijing Institute for General Artificial Intelligence（北京通用人工智能研究院）；Zhejiang University（浙江大学）；BUPT（北京邮电大学）
Stanford University（斯坦福大学）；École de Technologie Supérieure（高等技术学院）
Robotic manipulators navigating cluttered shelves or cabinets may find it challenging to avoid contact with obstacles. Indeed, rearranging obstacles may be necessary to access a target. Rather than planning explicit motions that place obstacles into a desired pose, we suggest allowing incidental contacts to rearrange obstacles while monitoring contacts for safety. Bypassing object identification, we present a method for categorizing object motions from tactile data collected from incidental contacts with a capacitive tactile skin on an Allegro Hand. We formalize tactile cues associated with ca…
NVIDIA（英伟达）；Georgia Institute of Technology（佐治亚理工学院）；University of Utah（犹他大学）
Using simulation to train robot manipulation policies holds the promise of an almost unlimited amount of training data, generated safely out of harm's way. One of the key challenges of using simulation, to date, has been to bridge the reality gap, so that policies trained in simulation can be deployed in the real world. We explore the reality gap in the context of learning a contextual policy for multi-fingered robotic grasping. We propose a Grasping Objects Approach for Tactile (GOAT) robotic hands, learning to overcome the reality gap problem. In our approach we use human hand motion demonst…
A*STAR Institute for Infocomm Research (IR)（A*STAR资讯通信研究院（IR））；A*STAR Institute of High Performance Computing (IHPC)（A*STAR高性能计算研究院（IHPC））
Robots are increasingly expected to manipulate objects in ever more unstructured environments where the object properties have high perceptual uncertainty from any single sensory modality. This directly impacts successful object manipulation. In this work, we propose a reinforcement learning-based motion planning framework for object manipulation which makes use of both on-the-fly multisensory feedback and a learned attention-guided deep affordance model as perceptual states. The affordance model is learned from multiple sensory modalities, including vision and touch (tactile and force/torque)…
Aalto University（阿尔托大学）；University of Amsterdam（阿姆斯特丹大学）；Centrum Wiskunde & Informatica（数学与计算机科学研究中心）；Bristol Robotics Laboratory（布里斯托尔机器人实验室）
Extracting and binding salient information from different sensory modalities to determine common features in the environment is a significant challenge in robotics. Here we present MuPNet (Multi-modal Predictive Coding Network), a biologically plausible network architecture for extracting joint latent features from visuo-tactile sensory data gathered from a biomimetic mobile robot. In this study we evaluate MuPNet applied to place recognition as a simulated biomimetic robot platform explores visually aliased environments. The F1 scores demonstrate that its performance over prior hand-crafted s…
