The object perception capabilities of humans are impressive, and this becomes even more evident when trying to develop solutions with a similar proficiency in autonomous robots. While there have been notable advancements in the technologies for artificial vision and touch, the effective integration of these two sensory modalities in robotic applications still needs to be improved, and several open challenges exist. Taking inspiration from how humans combine visual and haptic perception to perceive object properties and drive the execution of manual tasks, this article summarises the current state of the art of visuo-haptic object perception in robots. Firstly, the biological basis of human multimodal object perception is outlined. Then, the latest advances in sensing technologies and data collection strategies for robots are discussed. Next, an overview of the main computational techniques is presented, highlighting the main challenges of multimodal machine learning and presenting a few representative articles in the areas of robotic object recognition, peripersonal space representation and manipulation. Finally, informed by the latest advancements and open challenges, this article outlines promising new research directions.
Multimodal object recognition is still an emerging field. Thus, publicly available datasets are still rare and of small size. This dataset was developed to help fill this void and presents multimodal data for 63 objects with some visual and haptic ambiguity. The dataset contains visual, kinesthetic and tactile (audio/vibrations) data. To completely solve sensory ambiguity, sensory integration/fusion would be required. This report describes the creation and structure of the dataset. The first section explains the underlying approach used to capture the visual and haptic properties of the object…
Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory i…
Armstrong Atlantic State Univ.（阿姆斯特朗大西洋州立大学）；Ovidius University（奥维迪乌斯大学）
Surgeons must accomplish complex technical and intellectual tasks that can generate unexpected and serious challenges with little or no room for error. In the last decade, computer simulations have played an increasing role in surgical training, pre-operative planning, and biomedical research. Specifically, visuo-haptic simulations have been the focus of research to develop advanced e-Learning systems facilitating surgical training. The cost of haptic hardware was reduced through mass scale production and as haptics gained popularity in the gaming industry. Visuo-haptic simulations combine the…
Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation
Technische Universität Darmstadt（达姆施塔特工业大学）；German Research Center for AI (DFKI)（德国人工智能研究中心）；Hessian Centre for Artificial Intelligence (Hessian.AI)（黑森州人工智能中心）；Centre for Cognitive Science (CogSci)（认知科学中心）；Nagoya University（名古屋大学）
CTU in Prague（布拉格捷克理工大学）；Aalto University（阿尔托大学）；Czech Institute of Informatics, Robotics, and Cybernetics, CTU in Prague（布拉格捷克理工大学捷克信息学、机器人与控制论研究所）
Recent advancements in object shape completion have enabled impressive object reconstructions using only visual input. However, due to self-occlusion, the reconstructions have high uncertainty in the occluded object parts, which negatively impacts the performance of downstream robotic tasks such as grasping. In this work, we propose an active visuo-haptic shape completion method called Act-VH that actively computes where to touch the objects based on the reconstruction uncertainty. Act-VH reconstructs objects from point clouds and calculates the reconstruction uncertainty using IGR, a recent s…
Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning
The University of Hong Kong（香港大学）；Center for Transformative Garment Production（变革性服装生产中心）；Southern University of Science and Technology（南方科技大学）；Baidu Research（百度研究院）
Spacetime Optimal-Transport Attention for Visuo-Haptic Imitation Learning of Contact-Rich Manipulation
Robotics Research Centre, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore（机器人研究中心，机械与航空航天工程学院，南洋理工大学，新加坡）；Wings Robotics（Wings机器人）
Technical University of Munich（慕尼黑工业大学）；ABB Corporate Research（ABB集团研究院）；University of Campania Luigi Vanvitelli（坎帕尼亚大学路易吉·万维泰利）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
In this work, we introduce the problem of cross-modal visuo-tactile object recognition with robotic active exploration. With this term, we mean that the robot observes a set of objects with visual perception and, later on, it is able to recognize such objects only with tactile exploration, without having touched any object before. Using a machine learning terminology, in our application we have a visual training set and a tactile test set, or vice versa. To tackle this problem, we propose an approach constituted by four steps: finding a visuo-tactile common representation, defining a suitable …
University of Heidelberg（海德堡大学）；University of Genoa（热那亚大学）；University Niccolò Cusano（尼可洛·库萨诺大学）
Bimanual object manipulation involves multiple visuo-haptic sensory feedbacks arising from the interaction with the environment that are managed from the central nervous system and consequently translated in motor commands. Kinematic strategies that occur during bimanual coupled tasks are still a scientific debate despite modern advances in haptics and robotics. Current technologies may have the potential to provide realistic scenarios involving the entire upper limb extremities during multi-joint movements but are not yet exploited to their full potential. The present study explores how hands…
In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. An aspirational goal is to construct self-improving robots: robots that can learn and improve on their own, from autonomous interaction with minimal human supervision or oversight. Such robots could collect and train on much larger datasets, and thus learn more robust and performant policies. While reinforcement learning offers a framework for such autonomous learning via trial-and-error, practical realizations end up requiring extensive human supervision for reward f…
