Motion prediction in unstructured environments is a difficult problem and is essential for safe and efficient human-robot space sharing and collaboration. In this work, we focus on manipulation movements in environments such as homes, workplaces or restaurants, where the overall task and environment can be leveraged to produce accurate motion prediction. For these cases we propose an algorithmic framework that accounts explicitly for the environment geometry based on a model of affordances and a model of short-term human dynamics both trained on motion capture data. We propose dedicated function networks for graspability and placebility affordances and we make use of a dedicated RNN for short-term motion prediction. The prediction of grasp and placement probability densities are used by a constraint-based trajectory optimizer to produce a full-body motion prediction over the entire horizon. We show by comparing to ground truth data that we achieve similar performance for full-body motion predictions as using oracle grasp and place locations.
Shape informs how an object should be grasped, both in terms of where and how. As such, this paper describes a segmentation-based architecture for decomposing objects sensed with a depth camera into multiple primitive shapes, along with a post-processing pipeline for robotic grasping. Segmentation employs a deep network, called PS-CNN, trained on synthetic data with 6 classes of primitive shapes and generated using a simulation engine. Each primitive shape is designed with parametrized grasp families, permitting the pipeline to identify multiple grasp candidates per shape region. The grasps ar…
Indian Institute of Science（印度科学学院）；Tata Consultancy Services（塔塔咨询服务公司）
Aerial transportation requires a simple yet reliable gripper for picking and placing objects of interest. In this work, we design an aerial gripper for passive grasping and impulsive release of ferrous coated objects. Permanent magnets are used for passive grasping and the Scotch Yoke mechanism is used for providing impulsive force to drop the object. The load carrying capacity of the gripper is calculated theoretically and experimentally. The parameters such as the radius of the rotating disk and length of the slider in the Scotch Yoke mechanism were optimized using weighted geometric program…
The Grasp Reset Mechanism: An Automated Apparatus for Conducting Grasping Trials
Collaborative Robotics and Intelligent Systems (CoRIS) Institute（协作机器人与智能系统研究所）；Oregon State University（俄勒冈州立大学）；Southwest Research Institute (SwRI)（西南研究院）；UMass Lowell Nerve Center（马萨诸塞大学洛厄尔分校神经中心）
Lan-grasp: Using Large Language Models for Semantic Object Grasping and Placement
Department of Computer Science and Artificial Intelligence, University of Technology Nuremberg（计算机科学与人工智能系，图恩技术大学）；Department of Engineering ``Enzo Ferrari'' (DIEF), University of Modena and Reggio Emilia（恩佐·费尔拉蒂工程系（DIEF），摩德纳和雷吉奥艾米利亚大学）；Technical University of Munich（慕尼黑技术大学）
Robot learning provides a number of ways to teach robots simple skills, such as grasping. However, these skills are usually trained in open, clutter-free environments, and therefore would likely cause undesirable collisions in more complex, cluttered environments. In this work, we introduce an affordance model based on a graph representation of an environment, which is optimised during deployment to find suitable robot configurations to start a skill from, such that the skill can be executed without any collisions. We demonstrate that our method can generalise a priori acquired skills to previ…
Fraunhofer Institute for Manufacturing Engineering and Automation IPA（弗劳恩霍夫制造工程与自动化研究所（IPA））
In this paper, we introduce a novel learning-based approach for grasping known rigid objects in highly cluttered scenes and precisely placing them based on depth images. Our Placement Quality Network (PQ-Net) estimates the object pose and the quality for each automatically generated grasp pose for multiple objects simultaneously at 92 fps in a single forward pass of a neural network. All grasping and placement trials are executed in a physics simulation and the gained experience is transferred to the real world using domain randomization. We demonstrate that our policy successfully transfers t…
ARC Centre for Excellence for Robot Vision（澳大利亚研究理事会机器人视觉卓越中心）；Monash University（莫纳什大学）
Grasping is the dominant approach for robot manipulation, but only a single object can be grasped at a time. Nonprehensile manipulation offers richer set of interactions, however state-of-the-art is limited to using the end-effector only. We propose using a robot link (forearm) to push multiple objects at once. In a simulated task where the robot's task is to sort two kinds of objects into their respective goal regions, we show that a greedy strategy that uses a combination of forearm pushes and pick and place operations reduces task completion time by %28 compared to picking and placing each …
Fraunhofer IOSB（弗劳恩霍夫IOSB研究所）；Fraunhofer Institute of Optronics, System Technologies and Image Exploitation（弗劳恩霍夫光电、系统技术与图像开发研究所）
In order to enable physical human-robot interaction where humans and (mobile) manipulators share their workspace and work together, robots have to be equipped with important capabilities to guarantee human safety. The robots have to recognize possible collisions with the human co-worker and react anticipatorily by adapting their motion to avert dangerous situations while they are executing their task. Therefore, methods have been developed that allow to monitor the workspace of mobile manipulators using multiple depth sensors to gather information about the robot environment. This encompasses …
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Facebook AI Research（脸书人工智能研究院）；Technical University of Munich（慕尼黑工业大学）
Humans have impressive generalization capabilities when it comes to manipulating objects and tools in completely novel environments. These capabilities are, at least partially, a result of humans having internal models of their bodies and any grasped object. How to learn such body schemas for robots remains an open problem. In this work, we develop an self-supervised approach that can extend a robot's kinematic model when grasping an object from visual latent representations. Our framework comprises two components: (1) we present a multi-modal keypoint detector: an autoencoder architecture tra…
A generalist robot equipped with learned skills must be able to perform many tasks in many different environments. However, zero-shot generalization to new settings is not always possible. When the robot encounters a new environment or object, it may need to finetune some of its previously learned skills to accommodate this change. But crucially, previously learned behaviors and models should still be suitable to accelerate this relearning. In this paper, we aim to study how generative models of possible outcomes can allow a robot to learn visual representations of affordances, so that the rob…
