A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.
Many previous works approach vision-based robotic grasping by training a value network that evaluates grasp proposals. These approaches require an optimization process at run-time to infer the best action from the value network. As a result, the inference time grows exponentially as the dimension of action space increases. We propose an alternative method, by directly training a neural density model to approximate the conditional distribution of successful grasp poses from the input images. We construct a neural network that combines Gaussian mixture and normalizing flows, which is able to rep…
This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object's geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object's geometry by uti…
ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping
TAMS (Technical Aspects of Multimodal Systems), Department of Informatics, Universität Hamburg（乌尔姆大学信息学院技术多模态系统部门）；Agile Robots AG（敏捷机器人公司）；Technical University of Munich（慕尼黑技术大学）
Tactile perception is an essential ability of intelligent robots in interaction with their surrounding environments. This perception as an intermediate level acts between sensation and action and has to be defined properly to generate suitable action in response to sensed data. In this paper, we propose a feedback approach to address robot grasping task using force-torque tactile sensing. While visual perception is an essential part for gross reaching, constant utilization of this sensing modality can negatively affect the grasping process with overwhelming computation. In such case, human bei…
Institute for Infocomm Research, Agency for Science, Technology and Research（资讯通信研究院，科技研究局）
6D object pose estimation is widely applied in robotic tasks such as grasping and manipulation. Prior methods using RGB-only images are vulnerable to heavy occlusion and poor illumination, so it is important to complement them with depth information. However, existing methods using RGB-D data cannot adequately exploit consistent and complementary information between RGB and depth modalities. In this paper, we present a novel method to effectively consider the correlation within and across both modalities with attention mechanism to learn discriminative and compact multi-modal features. Then, e…
Optimizing Grasping in Legged Robots: A Deep Learning Approach to Loco-Manipulation
College of Mechanical Engineering, Federal University of Uberlândia（联邦大学伯南迪亚学院机械工程学院）；Department of Mechanical Engineering, University of São Paulo（圣保罗大学机械工程系）
National Technical University of Athens（雅典国家技术大学）；Technische Universität Darmstadt（达姆施塔特工业大学）；Carnegie Mellon University（卡内基梅隆大学）
Inherent morphological characteristics in objects may offer a wide range of plausible grasping orientations that obfuscates the visual learning of robotic grasping. Existing grasp generation approaches are cursed to construct discontinuous grasp maps by aggregating annotations for drastically different orientations per grasping point. Moreover, current methods generate grasp candidates across a single direction in the robot's viewpoint, ignoring its feasibility constraints. In this paper, we propose a novel augmented grasp map representation, suitable for pixel-wise synthesis, that locally dis…
MultiGraspNet: A Multitask 3D Vision Model for Multi-gripper Robotic Grasping
VANDAL Laboratory, Department of Control and Computer Engineering, Politecnico di Torino（沃纳达实验室，控制与计算机工程系，都灵理工大学）；Comau S.p.A., Advanced Automation Solutions（Comau S.p.A.，先进自动化解决方案）；Istituto Italiano di Tecnologia（意大利技术研究所）
Whole-Body Proprioceptive Morphing: A Modular Soft Gripper for Robust Cross-Scale Grasping
Mechanical Engineering Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机械工程系）；Robotics Department at the University of Michigan-Ann Arbor（密歇根大学安娜堡分校机器人学系）
LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion
school of Data Science, Fudan University（复旦大学数据科学学院）；Fudan University（复旦大学）；School of statistics and information, Shanghai University of International Business and Economics（上海对外经贸大学统计与信息学院）
CLASP: Closed-loop Asynchronous Spatial Perception for Open-vocabulary Desktop Object Grasping
Harbin Institute of Technology（哈尔滨工业大学）；National Key Laboratory of Smart Farm Technologies and Systems（智慧农场技术与系统全国重点实验室）；Peng Cheng Laboratory（鹏城实验室）；Northeastern University（东北大学）
The University of Tokyo（东京大学）；Preferred Networks, Inc.（Preferred Networks 公司）
For in-hand manipulation, estimation of the object pose inside the hand is one of the important functions to manipulate objects to the target pose. Since in-hand manipulation tends to cause occlusions by the hand or the object itself, image information only is not sufficient for in-hand object pose estimation. Multiple modalities can be used in this case, the advantage is that other modalities can compensate for occlusion, noise, and sensor malfunctions. Even though deciding the utilization rate of a modality (referred to as reliability value) corresponding to the situations is important, the …
ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers
Chair of Cyber-Physical Systems（感知系统教授席位）；Institute of Machine Learning and Neural Computation（机器学习与神经计算研究所）；Technical University of Leoben（莱比锡技术大学）；Graz University of Technology（格拉茨技术大学）
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Facebook AI Research（脸书人工智能研究院）；Technical University of Munich（慕尼黑工业大学）
Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body schemas for robots remains an open problem. In this work, we develop an self-supervised approach that can extend a robot's kinematic model when grasping an object from visual latent representations. Our framework comprises two components: (1) we present a multi-modal keypoint detector: an autoencoder architecture tra…
Humans usually perceive the world in a multimodal way that vision, touch, sound are utilised to understand surroundings from various dimensions. These senses are combined together to achieve a synergistic effect where the learning is more effectively than using each sense separately. For robotics, vision and touch are two key senses for the dexterous manipulation. Vision usually gives us apparent features like shape, color, and the touch provides local information such as friction, texture, etc. Due to the complementary properties between visual and tactile senses, it is desirable for us to co…
TU Kaiserslautern（凯泽斯劳滕工业大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. This offers new possibilities for bare handed interaction within virtual environments. This work proposes a hand gesture authoring tool for object specific grab gestures allowing virtual objects to be grabbed as in the real world. The presented solution uses template matching for gesture recognition and requires no technical knowledge to design…
Assistive robotic systems endeavour to support those with movement disabilities, enabling them to move again and regain functionality. Main issue with these systems is the complexity of their low-level control, and how to translate this to simpler, higher level commands that are easy and intuitive for a human user to interact with. We have created a multi-modal system, consisting of different sensing, decision making and actuating modalities, leading to intuitive, human-in-the-loop assistive robotics. The system takes its cue from the user's gaze, to decode their intentions and implement low-l…
Look-to-Touch: A Vision-Enhanced Proximity and Tactile Sensor for Distance and Geometry Perception in Robotic Manipulation
State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University（上海交通大学机械系统与振动国家重点实验室）；Robotics Institute, School of Mechanical Engineering, Shanghai Jiao Tong University（上海交通大学机械工程学院机器人研究所）
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（坎帕尼亚路易吉·凡维泰利大学工程系）；Institute for Artificial Intelligence, Universität Bremen（不来梅大学人工智能研究所）
Manipulation planning and control are relevant building blocks of a robotic system and their tight integration is a key factor to improve robot autonomy and allows robots to perform manipulation tasks of increasing complexity, such as those needed in the in-store logistics domain. Supermarkets contain a large variety of objects to be placed on the shelf layers with specific constraints, doing this with a robot is a challenge and requires a high dexterity. However, an integration of reactive grasping control and motion planning can allow robots to perform such tasks even with grippers with limi…
University of Maryland（马里兰大学）；Institute for Advanced Computer Studies（高级计算机研究所）
Grasping objects requires tight integration between visual and tactile feedback. However, there is an inherent difference in the scale at which both these input modalities operate. It is thus necessary to be able to analyze tactile feedback in isolation in order to gain information about the surface the end-effector is operating on, such that more fine-grained features may be extracted from the surroundings. For tactile perception of the robot, inspired by the concept of the tactile flow in humans, we present the computational tactile flow to improve the analysis of the tactile feedback in rob…
Robotic Perception with a Large Tactile-Vision-Language Model for Physical Property Inference
College of Big Data and Internet, Shenzhen Technology University, China（大数据与互联网学院，深圳科技大学，中国）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University, China（中德智能制造学院，深圳科技大学，中国）；Robotics Department, Institute of Smart Systems and Artificial Intelligence (ISSAI), Nazarbayev University, Kazakhstan（机器人系，智能系统与人工智能研究所（ISSAI），纳扎尔巴耶夫大学，哈萨克斯坦）；Foundation Robotics Labs, Germany（基础机器人实验室，德国）
AnyViewDex: View-Invariant Dexterous Manipulation from RGB Observations
Robotics Research Center, IIIT-H（IIIT-H机器人研究中心）；Veermata Jijabai Technological Institute Mumbai（孟买维尔马塔·吉贾拜技术学院）；Indian Institute of Science Education and Research Bhopal（印度科学教育与研究学院博帕尔分校）
Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to…
