Developing personal robots that can perform a diverse range of manipulation tasks in unstructured environments necessitates solving several challenges for robotic grasping systems. We take a step towards this broader goal by presenting the first RL-based system, to our knowledge, for a mobile manipulator that can (a) achieve targeted grasping generalizing to unseen target objects, (b) learn complex grasping strategies for cluttered scenes with occluded objects, and (c) perform active vision through its movable wrist camera to better locate objects. The system is informed of the desired target object in the form of a single, arbitrary-pose RGB image of that object, enabling the system to generalize to unseen objects without retraining. To achieve such a system, we combine several advances in deep reinforcement learning and present a large-scale distributed training system using synchronous SGD that seamlessly scales to multi-node, multi-GPU infrastructure to make rapid prototyping easier. We train and evaluate our system in a simulated environment, identify key components for improving performance, analyze its behaviors, and transfer to a real-world setup.
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Heriot-Watt University（赫瑞-瓦特大学）
Agile control of mobile manipulator is challenging because of the high complexity coupled by the robotic system and the unstructured working environment. Tracking and grasping a dynamic object with a random trajectory is even harder. In this paper, a multi-task reinforcement learning-based mobile manipulation control framework is proposed to achieve general dynamic object tracking and grasping. Several basic types of dynamic trajectories are chosen as the task training set. To improve the policy generalization in practice, random noise and dynamics randomization are introduced during the train…
In this paper, we investigate the problem of grasping novel objects in unstructured environments. To address this problem, consideration of the object geometry, reachability and force closure analysis are required. We propose a framework for grasping unknown objects by localizing contact regions on the contours formed by a set of depth edges in a single view 2D depth image. According to the edge geometric features obtained from analyzing the data of the depth map, the contact regions are determined. Finally,We validate the performance of the approach by applying it to the scenes with both sing…
Huazhong University of Science and Technology（华中科技大学）；South East University（东南大学）
To realize a robust robotic grasping system for unknown objects in an unstructured environment, large amounts of grasp data and 3D model data for the object are required, the sizes of which directly affect the rate of successful grasps. To reduce the time cost of data acquisition and labeling and increase the rate of successful grasps, we developed a self-supervised learning mechanism to control grasp tasks performed by manipulators. First, a manipulator automatically collects the point cloud for the objects from multiple perspectives to increase the efficiency of data acquisition. The complet…
The Grasp Reset Mechanism: An Automated Apparatus for Conducting Grasping Trials
Collaborative Robotics and Intelligent Systems (CoRIS) Institute（协作机器人与智能系统研究所）；Oregon State University（俄勒冈州立大学）；Southwest Research Institute (SwRI)（西南研究院）；UMass Lowell Nerve Center（马萨诸塞大学洛厄尔分校神经中心）
This paper studies robustness in planar grasping from a geometric perspective. By treating grasping as a process that shapes the free-space of an object over time, we can define three types of certificates to guarantee success of a grasp: (a) invariance under an initial set, (b) convergence towards a goal grasp, and (c) observability over the final object pose. We develop convex-combinatorial models for each of these certificates, which can be expressed as simple semi-algebraic relations under mild-modeling assumptions. By leveraging these models to synthesize certificates, we optimize certifi…
Vision-Guided Targeted Grasping and Vibration for Robotic Pollination in Controlled Environments
Department of Artificial Intelligence, Korea University（韩国大学人工智能系）；University of California, Los Angeles（加州大学洛杉矶分校）；Department of Mechanical & Aerospace Engineering, University of California, Los Angeles（加州大学洛杉矶分校机械与航空航天工程系）；Department of Computer Science, University of California, Los Angeles（加州大学洛杉矶分校计算机科学系）
We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real …
Grasping objects of different shapes and sizes - a foundational, effortless skill for humans - remains a challenging task in robotics. Although model-based approaches can predict stable grasp configurations for known object models, they struggle to generalize to novel objects and often operate in a non-interactive open-loop manner. In this work, we present a reinforcement learning framework that learns the interactive grasping of various geometrically distinct real-world objects by continuously controlling an anthropomorphic robotic hand. We explore several explicit representations of object g…
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
This paper uses a mobile manipulator with a collaborative robotic arm to manipulate objects beyond the robot's maximum payload. It proposes a single-shot probabilistic roadmap-based method to plan and optimize manipulation motion with environment support. The method uses an expanded object mesh model to examine contact and randomly explores object motion while keeping contact and securing affordable grasping force. It generates robotic motion trajectories after obtaining object motion using an optimization-based algorithm. With the proposed method's help, we can plan contact-rich manipulation …
Robotic Grasping of Harvested Tomato Trusses Using Vision and Online Learning
Delft University of Technology（代尔夫特理工大学）；Czech Technical University in Prague（布拉格捷克技术大学）；Czech Institute of Informatics, Robotics, and Cybernetics（捷克信息学、机器人学与控制论研究所）
University of Birmingham（伯明翰大学）；Sorbonne Université（索邦大学）
This paper presents a new dual quaternion-based formulation for pose-based visual servoing. Extending our previous work on local contact moment (LoCoMo) based grasp planning, we demonstrate grasping of arbitrarily moving objects in 3D space. Instead of using the conventional axis-angle parameterization, dual quaternions allow designing the visual servoing task in a more compact manner and provide robustness to manipulator singularities. Given an object point cloud, LoCoMo generates a ranked list of grasp and pre-grasp poses, which are used as desired poses for visual servoing. Whenever the obj…
Language-Guided Grasping under Partial Observation for Mobile Manipulation in Field Inspection and Maintenance
Department of Mechanical Engineering, Federal University of Uberlândia（联邦大学伯南迪利亚机械工程系）；Department of Mechanical Engineering, University of São Paulo（圣保罗大学机械工程系）；Graduate School of Information Sciences, Tohoku University（东北大学信息科学研究生院）；Faculdade Israelita de Ensino e Pesquisa Albert Einstein, Hospital Israelita Albert Einstein（艾伯特·爱因斯坦以色列教学与研究学院，艾伯特·爱因斯坦医院）
University of Nevada, Reno（内华达大学雷诺分校）；Swansea University（斯旺西大学）
Robot manipulation and grasping mechanisms have received considerable attention in the recent past, leading to the development of wide range of industrial applications. This paper proposes the development of an autonomous robotic grasping system for object sorting application. RGB-D data is used by the robot for performing object detection, pose estimation, trajectory generation, and object sorting tasks. The proposed approach can also handle grasping certain objects chosen by users. Trained convolutional neural networks are used to perform object detection and determine the corresponding poin…
Fraunhofer IOSB（弗劳恩霍夫IOSB研究所）；Fraunhofer Institute of Optronics, System Technologies and Image Exploitation（弗劳恩霍夫光电、系统技术与图像开发研究所）
In order to enable physical human-robot interaction where humans and (mobile) manipulators share their workspace and work together, robots have to be equipped with important capabilities to guarantee human safety. The robots have to recognize possible collisions with the human co-worker and react anticipatorily by adapting their motion to avert dangerous situations while they are executing their task. Therefore, methods have been developed that allow to monitor the workspace of mobile manipulators using multiple depth sensors to gather information about the robot environment. This encompasses …
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
Aiming at the traditional grasping method for manipulators based on 2D camera, when faced with the scene of gathering or covering, it can hardly perform well in unstructured scenes that appear as gathering and covering, for the reason that can't recognize objects accurately in cluster scenes from a single perspective and the manipulators can't make the environment better for grasping. In this case, a novel method of pushing-grasping collaborative based on the deep Q-network in dual perspectives is proposed in this paper. This method adopts an improved deep Q network algorithm, with an RGB-D ca…
Czech Technical University in Prague（布拉格捷克理工大学）；New York University（纽约大学）；University of Pennsylvania（宾夕法尼亚大学）
Mobile manipulators have the potential to revolutionize modern agriculture, logistics and manufacturing. In this work, we present the design of a ground-based mobile manipulator for automated structure assembly. The proposed system is capable of autonomous localization, grasping, transportation and deployment of construction material in a semi-structured environment. Special effort was put into making the system invariant to lighting changes, and not reliant on external positioning systems. Therefore, the presented system is self-contained and capable of operating in outdoor and indoor conditi…
We present the design and implementation of a taskable reactive mobile manipulation system. In contrary to related work, we treat the arm and base degrees of freedom as a holistic structure which greatly improves the speed and fluidity of the resulting motion. At the core of this approach is a robust and reactive motion controller which can achieve a desired end-effector pose, while avoiding joint position and velocity limits, and ensuring the mobile manipulator is manoeuvrable throughout the trajectory. This can support sensor-based behaviours such as closed-loop visual grasping. As no planni…
Visual-tactile manipulation to collect household waste in outdoor
Instituto Universitario de Investigación Informática, Universidad de Alicante（阿利坎特大学大学信息学研究学院）；Universidad de Alicante（阿利坎特大学）；Universidad Miguel Hernández（米格尔·埃尔南德斯大学）
Quasi-static models of robotic motion with frictional contact provide a computationally efficient framework for analysis and have been widely used for planning and control of non-prehensile manipulation. In this work, we present a novel quasi-static model of planar manipulation that directly maps commanded manipulator velocities to object motion. While quasi-static models have traditionally been unable to capture grasping and jamming behaviors, our approach solves this issue by explicitly modeling the limiting behavior of a velocity-controlled manipulator. We retain the precise modeling of sur…
Guangdong University of Technology（广东工业大学）；South China University of Technology（华南理工大学）
Humans can accurately determine whether the object in hand has slipped or not by visual and tactile perception. However, it is still a challenge for robots to detect in-hand object slip through visuo-tactile fusion. To address this issue, a novel visuo-tactile fusion deep neural network is proposed to detect slip, which is a time-dependent continuous action. By using the multi-scale temporal convolution network (MS-TCN) to extract the temporal features of visual and tactile data, the slip can be detected effectively. In this paper, a 7-dregree-of-freedom (7-DoF) robot manipulator equipped with…
Queensland University of Technology（昆士兰科技大学）；LYRO Robotics（LYRO机器人公司）
We present a generalised architecture for reactive mobile manipulation while a robot's base is in motion toward the next objective in a high-level task. By performing tasks on-the-move, overall cycle time is reduced compared to methods where the base pauses during manipulation. Reactive control of the manipulator enables grasping objects with unpredictable motion while improving robustness against perception errors, environmental disturbances, and inaccurate robot control compared to open-loop, trajectory-based planning approaches. We present an example implementation of the architecture and i…
Supertoroid fitting of objects with holes for robotic grasping and scene generation
Polythechnic University of Catalonia (UPC)（加泰罗尼亚理工大学（UPC））；Locus Robotics（洛库斯机器人公司）；Nauticus Robotics（诺蒂库斯机器人公司）；Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（机器人与信息自动化研究所（CSIC-UPC））
Hook-Based Aerial Payload Grasping from a Moving Platform
Systems and Control Lab, HUN-REN Institute for Computer Science and Control（系统与控制实验室，HUN-REN计算机科学与控制研究所）；National Laboratory for Autonomous Systems（自主系统国家实验室）；Control Systems Group of the Eindhoven University of Technology（埃因霍温理工大学控制系统组）
DynaMOMA: Instantaneous Prediction of Grasp Poses for Mobile Manipulation of Dynamic Objects
National University of Defense Technology（国防科技大学）；CFCS, Peking University（北京大学前沿计算研究中心）；Defense Innovation Institute, Academy of Military Sciences（军事科学院国防科技创新研究院）；Wuhan University（武汉大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院产业人工智能研究院）
Improbable AI Lab（Improbable人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；University of Washington（华盛顿大学）
We study the problem of object retrieval in scenarios where visual sensing is absent, object shapes are unknown beforehand and objects can move freely, like grabbing objects out of a drawer. Successful solutions require localizing free objects, identifying specific object instances, and then grasping the identified objects, only using touch feedback. Unlike vision, where cameras can observe the entire scene, touch sensors are local and only observe parts of the scene that are in contact with the manipulator. Moreover, information gathering via touch sensors necessitates applying forces on the …
We present a proprioceptive teleoperation system that uses a reflexive grasping algorithm to enhance the speed and robustness of pick-and-place tasks. The system consists of two manipulators that use quasi-direct-drive actuation to provide highly transparent force feedback. The end-effector has bimodal force sensors that measure 3-axis force information and 2-dimensional contact location. This information is used for anti-slip and re-grasping reflexes. When the user makes contact with the desired object, the re-grasping reflex aligns the gripper fingers with antipodal points on the object to m…
The vast majority of visual animals actively control their eyes, heads, and/or bodies to direct their gaze toward different parts of their environment. In contrast, recent applications of reinforcement learning in robotic manipulation employ cameras as passive sensors. These are carefully placed to view a scene from a fixed pose. Active perception allows animals to gather the most relevant information about the world and focus their computational resources where needed. It also enables them to view objects from different distances and viewpoints, providing a rich visual experience from which t…
Forestry machines are heavy vehicles performing complex manipulation tasks in unstructured production forest environments. Together with the complex dynamics of the on-board hydraulically actuated cranes, the rough forest terrains have posed a particular challenge in forestry automation. In this study, the feasibility of applying reinforcement learning control to forestry crane manipulators is investigated in a simulated environment. Our results show that it is possible to learn successful actuator-space control policies for energy efficient log grasping by invoking a simple curriculum in a de…
In this paper, we present the design, simulation and experimental validation of a control architecture for a flying hand, i.e., a system made of an unmanned aerial vehicle, a robotic manipulator and a gripper, which is grasping an object fixed on a vertical wall. The goal of this work is to show that the overall control allows the flying hand to approach the wall, to dock on the object by means of the gripper, take the object and fly away. The control strategy has been implemented and validated in the simulated model and in experiments on the complete flying hand system.
University of Pennsylvania（宾夕法尼亚大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
In this paper we address mobile manipulation planning problems in the presence of sensing and environmental uncertainty. In particular, we consider mobile sensing manipulators operating in environments with unknown geometry and uncertain movable objects, while being responsible for accomplishing tasks requiring grasping and releasing objects in a logical fashion. Existing algorithms either do not scale well or neglect sensing and/or environmental uncertainty. To face these challenges, we propose a hybrid control architecture, where a symbolic controller generates high-level manipulation comman…
Dipartimento di Ingegneria, Università degli Studi della Campania Luigi Vanvitelli（坎帕尼亚路易吉·凡维泰利大学工程系）；Institute for Artificial Intelligence, Universität Bremen（不来梅大学人工智能研究所）
Manipulation planning and control are relevant building blocks of a robotic system and their tight integration is a key factor to improve robot autonomy and allows robots to perform manipulation tasks of increasing complexity, such as those needed in the in-store logistics domain. Supermarkets contain a large variety of objects to be placed on the shelf layers with specific constraints, doing this with a robot is a challenge and requires a high dexterity. However, an integration of reactive grasping control and motion planning can allow robots to perform such tasks even with grippers with limi…
