Active Visuo-Tactile Point Cloud Registration for Accurate Pose Estimation of Objects in an Unknown Workspace
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；Technische Universität München（慕尼黑工业大学）；Donders Institute for Brain and Cognition（唐德斯脑与认知研究所）；Radboud University（拉德堡德大学）
Prajval Kumar Murali, Michael Gentner, Mohsen Kaboli
备注：
Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021)
This paper proposes a novel active visuo-tactile based methodology wherein the accurate estimation of the time-invariant SE(3) pose of objects is considered for autonomous robotic manipulators. The robot equipped with tactile sensors on the gripper is guided by a vision estimate to actively explore and localize the objects in the unknown workspace. The robot is capable of reasoning over multiple potential actions, and execute the action to maximize information gain to update the current belief of the object. We formulate the pose estimation process as a linear translation invariant quaternion filter (TIQF) by decoupling the estimation of translation and rotation and formulating the update and measurement model in linear form. We perform pose estimation sequentially on acquired measurements using very sparse point cloud as acquiring each measurement using tactile sensing is time consuming. Furthermore, our proposed method is computationally efficient to perform an exhaustive uncertainty-based active touch selection strategy in real-time without the need for trading information gain with execution time. We evaluated the performance of our approach extensively in simulation and by a robotic system.
Estimation of tactile properties from vision, such as slipperiness or roughness, is important to effectively interact with the environment. These tactile properties help us decide which actions we should choose and how to perform them. E.g., we can drive slower if we see that we have bad traction or grasp tighter if an item looks slippery. We believe that this ability also helps robots to enhance their understanding of the environment, and thus enables them to tailor their actions to the situation at hand. We therefore propose a model to estimate the degree of tactile properties from visual pe…
Georgia Institute of Technology（佐治亚理工学院）；NVIDIA（英伟达）；University of Utah（犹他大学）；University of Washington（华盛顿大学）
To perform complex tasks, robots must be able to interact with and manipulate their surroundings. One of the key challenges in accomplishing this is robust state estimation during physical interactions, where the state involves not only the robot and the object being manipulated, but also the state of the contact itself. In this work, within the context of planar pushing, we extend previous inference-based approaches to state estimation in several ways. We estimate the robot, object, and the contact state on multiple manipulation platforms configured with a vision-based articulated model track…
Samsung AI Center Montreal（三星蒙特利尔人工智能中心）；York University（约克大学）；McGill University（麦吉尔大学）；Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）
We introduce a new class of vision-based sensor and associated algorithmic processes that combine visual imaging with high-resolution tactile sending, all in a uniform hardware and computational architecture. We demonstrate the sensor's efficacy for both multi-modal object recognition and metrology. Object recognition is typically formulated as an unimodal task, but by combining two sensor modalities we show that we can achieve several significant performance improvements. This sensor, named the See-Through-your-Skin sensor (STS), is designed to provide rich multi-modal sensing of contact surf…
When Vision Meets Touch: A Contemporary Review for Visuotactile Sensors from the Signal Processing Perspective
Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；RISC-V International Open Source Laboratory（RISC-V国际开源实验室）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）
Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions
The BioRobotics Institute, Sant’Anna School of Advanced Studies（圣安娜高级研究院生物机器人研究所）；Departement of Excellence in Robotics & AI, Sant’Anna School of Advanced Studies（圣安娜高级研究院机器人与人工智能卓越系）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）
UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking
ScaleLab, Shanghai Jiao Tong University（上海交通大学ScaleLab）；D-Robotics；ViTai Robotics；The University of Hong Kong（香港大学）；Nanjing University（南京大学）；Shenzhen University（深圳大学）；Wuhan University（武汉大学）；Fudan University（复旦大学）；Tsinghua University（清华大学）
Deformable objects manipulation can benefit from representations that seamlessly integrate vision and touch while handling occlusions. In this work, we present a novel approach for, and real-world demonstration of, multimodal visuo-tactile state-estimation and dynamics prediction for deformable objects. Our approach, VIRDO++, builds on recent progress in multimodal neural implicit representations for deformable object state-estimation [1] via a new formulation for deformation dynamics and a complementary state-estimation algorithm that (i) maintains a belief over deformations, and (ii) enables…
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）
Humans can accurately determine whether the object in hand has slipped or not by visual and tactile perception. However, it is still a challenge for robots to detect in-hand object slip through visuo-tactile fusion. To address this issue, a novel visuo-tactile fusion deep neural network is proposed to detect slip, which is a time-dependent continuous action. By using the multi-scale temporal convolution network (MS-TCN) to extract the temporal features of visual and tactile data, the slip can be detected effectively. In this paper, a 7-dregree-of-freedom (7-DoF) robot manipulator equipped with…
Facebook AI Research（Facebook人工智能研究院）；McGill University（麦吉尔大学）；University of California, Berkeley（加州大学伯克利分校）
Humans build 3D understandings of the world through active object exploration, using jointly their senses of vision and touch. However, in 3D shape reconstruction, most recent progress has relied on static datasets of limited sensory data such as RGB images, depth maps or haptic readings, leaving the active exploration of the shape largely unexplored. Inactive touch sensing for 3D reconstruction, the goal is to actively select the tactile readings that maximize the improvement in shape reconstruction accuracy. However, the development of deep learning-based active touch models is largely limit…
In this work, we propose a novel, bio-inspired multi-sensory SLAM approach called ViTa-SLAM. Compared to other multisensory SLAM variants, this approach allows for a seamless multi-sensory information fusion whilst naturally interacting with the environment. The algorithm is empirically evaluated in a simulated setting using a biomimetic robot platform called the WhiskEye. Our results show promising performance enhancements over existing bio-inspired SLAM approaches in terms of loop-closure detection.
OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation
TARS Robotics（TARS机器人实验室）；National University of Singapore（新加坡国立大学）；Fudan University（复旦大学）；CASIA（中国科学院自动化研究所）；Tsinghua University（清华大学）；Zhongguancun Academy（中关村学院）；Beihang University（北京航空航天大学）
Cross-Modal Visuo-Tactile Representation Learning with Action Chunking Transformers for Contact-Rich Manipulation
Guangdong Institute of Intelligence Science and Technology（广东智能科学与技术研究院）；Innovation and Research and Development Department, BoardWare Information System Company Ltd.（创新与研发部，BoardWare信息系统有限公司）；School of Artificial Intelligence, Nanjing Agricultural University（人工智能学院，南京农业大学）；School of Computing, National University of Singapore（计算机学院，新加坡国立大学）；School of Electronic Engineering and Automation, Guilin University of Electronic Technology（电子工程与自动化学院，桂林电子科技大学）
TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model
Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；School of Information, Renmin University of China（中国人民大学信息学院）；Fudan University（复旦大学）；TARS Robotics
OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework
Fudan University（复旦大学）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身智能重点实验室）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）；Neote AI；China Unicom（中国联通）
BMW Group（宝马集团）；University of Glasgow（格拉斯哥大学）；The Imperial College of Science, Technology and Medicine（帝国理工学院）；Donders Institute for Brain and Cognition, Radboud University（拉德堡德大学唐德斯脑认知研究所）
This work presents a novel active visuo-tactile based framework for robotic systems to accurately estimate pose of objects in dense cluttered environments. The scene representation is derived using a novel declutter graph (DG) which describes the relationship among objects in the scene for decluttering by leveraging semantic segmentation and grasp affordances networks. The graph formulation allows robots to efficiently declutter the workspace by autonomously selecting the next best object to remove and the optimal action (prehensile or non-prehensile) to perform. Furthermore, we propose a nove…
FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation
Shanghai Innovation Institute（上海创新研究院）；The University of Hong Kong（香港大学）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；Shanghai University（上海大学）
Self-Supervised Visuo-Tactile Pretraining to Locate and Follow Garment Features
UC Berkeley（加州大学伯克利分校）；TU Dresden（德累斯顿工业大学）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI)（人在环触觉互联网中心（CeTI））；Carnegie Mellon University（卡内基梅隆大学）
Many manipulation tasks require the robot to control the contact between a grasped compliant tool and the environment, e.g. scraping a frying pan with a spatula. However, modeling tool-environment interaction is difficult, especially when the tool is compliant, and the robot cannot be expected to have the full geometry and physical properties (e.g., mass, stiffness, and friction) of all the tools it must use. We propose a framework that learns to predict the effects of a robot's actions on the contact between the tool and the environment given visuo-tactile perception. Key to our framework is …
Visuo-Tactile Exploration of Unknown Rigid 3D Curvatures by Vision-Augmented Unified Force-Impedance Control
Chair of Robotics and Systems Intelligence, MIRMI-Munich Institute of Robotics and Machine Intelligence, Technical University of Munich（慕尼黑工业大学机器人与系统智能教席，慕尼黑机器人与机器智能研究所）；Centre for Tactile Internet with Human-in-the-Loop (CeTI)（以人为中心的触觉互联网中心）
Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning
Center on Frontiers of Computing Studies, School of Computer Science, Peking University（前沿计算研究中心，计算机科学学院，北京大学）；PKU-Agibot Lab, School of Computer Science, Peking University（北京大学计算机科学学院PKU-Agibot实验室）；National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（国家多媒体信息处理重点实验室，计算机科学学院，北京大学）
Active Cross-Modal Visuo-Tactile Perception of Deformable Linear Objects
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（工程学院，坎帕尼亚路易吉·范维蒂利大学）；Dipartimento di Ingegneria dell’Informazione, Università degli Studi di Padova（信息工程学院，帕多瓦大学）
Aalto University（阿尔托大学）；University of Bristol（布里斯托大学）；University of the West of England（西英格兰大学）
RatSLAM is a rat hippocampus-inspired visual Simultaneous Localization and Mapping (SLAM) framework capable of generating semi-metric topological representations of indoor and outdoor environments. Whisker-RatSLAM is a 6D extension of the RatSLAM and primarily focuses on object recognition by generating point clouds of objects based on whisking information. This paper introduces a novel extension to both former works that is referred to as ViTa-SLAM that harnesses both vision and tactile information for performing SLAM. This not only allows the robot to perform natural interaction with the env…
Xi’an-Jiaotong Liverpool University（西交利物浦大学）；The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；Southern University of Science and Technology（南方科技大学）
We propose a deep visuo-tactile model for realtime estimation of the liquid inside a deformable container in a proprioceptive way.We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations.The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolut…
TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance
Shanghai Jiao Tong University（上海交通大学）；Xense Robotics（Xense机器人公司）；Sun Yat-sen University（中山大学）；Oxford（牛津大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；UCSD（加州大学圣地亚哥分校）
Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \cite{dosovitskiy2021image} to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus at…
Grasping Deformable Objects via Reinforcement Learning with Cross-Modal Attention to Visuo-Tactile Inputs
Dept. of Electronic Engineering at Sogang University（ソガン大学电子工程系）；Dept. of Immersive Media and Engineering at Sungkyunkwan University（顺天大学沉浸媒体与工程系）；College of Medicine, Yonsei University（延世大学医学院）
