Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images
Ben-Gurion University of the Negev（内盖夫本-古里安大学）
Yuval Litvak, Armin Biess, Aharon Bar-Hillel
备注：
8 pages, 5 figures. This work has been accepted to the International Conference on Robotics and Automation (ICRA 2019). For associated video, see https://youtu.be/uMvq2-Tg-9g
Most of industrial robotic assembly tasks today require fixed initial conditions for successful assembly. These constraints induce high production costs and low adaptability to new tasks. In this work we aim towards flexible and adaptable robotic assembly by using 3D CAD models for all parts to be assembled. We focus on a generic assembly task - the Siemens Innovation Challenge - in which a robot needs to assemble a gear-like mechanism with high precision into an operating system. To obtain the millimeter-accuracy required for this task and industrial settings alike, we use a depth camera mounted near the robot end-effector. We present a high-accuracy two-stage pose estimation procedure based on deep convolutional neural networks, which includes detection, pose estimation, refinement, and handling of near- and full symmetries of parts. The networks are trained on simulated depth images with means to ensure successful transfer to the real robot. We obtain an average pose estimation error of 2.16 millimeters and 0.64 degree leading to 91% success rate for robotic assembly of randomly distributed parts. To the best of our knowledge, this is the first time that the Siemens Innovation Challenge is fully addressed, with all the parts assembled with high success rates.
OMRON SINIC X Corp.（欧姆龙SINIC X公司）；OMRON Corp.（欧姆龙公司）；Osaka University（大阪大学）；SenseTime Japan Ltd.（商汤日本有限公司）；National Inst. of Advanced Industrial Science and Technology (AIST)（国立先进工业科学技术研究所（AIST））；Laboro.AI Inc.（Laboro.AI公司）
We propose a versatile robotic system for kitting and assembly tasks which uses no jigs or commercial tool changers. Instead of specialized end effectors, it uses its two-finger grippers to grasp and hold tools to perform subtasks such as screwing and suctioning. A third gripper is used as a precision picking and centering tool, and uses in-built passive compliance to compensate for small position errors and uncertainty. A novel grasp point detection for bin picking is described for the kitting task, using a single depth map. Using the proposed system we competed in the Assembly Challenge of t…
Scalable Unseen Objects 6-DoF Absolute Pose Estimation with Robotic Integration
National Engineering Research Center for Robot Visual Perception and Control Technology, School of Artificial Intelligence and Robotics, Hunan University（机器人视觉感知与控制技术国家工程研究中心，人工智能与机器人学院，湖南大学）；School of Architecture and Art, Central South University（建筑与艺术学院，中南大学）；School of Computing and Communications, Lancaster University（计算与通讯学院，兰卡斯特大学）；Department of Computer Science and Software Engineering, The University of Western Australia（计算机科学与软件工程系，西澳大学）；School of Electrical and Electronic Engineering, Nanyang Technological University（电子与电气工程学院，南洋理工大学）
Benchmarking the Effects of Object Pose Estimation and Reconstruction on Robotic Grasping Success
Faculty of Electrical Engineering, Czech Technical University in Prague（捷克技术大学布拉格电子工程系）；Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与自动化研究所）
Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure …
In this work, we propose a framework called Auto-Assembly for automated robotic assembly from design files and demonstrate a practical implementation on modular parts joined by fastening using a robotic cell consisting of two robots. We show the flexibility of the approach by testing it on different input designs. Auto-Assembly consists of several parts: design analysis, assembly sequence generation, bill-of-process (BOP) generation, conversion of the BOP to control code, path planning, simulation, and execution of the control code to assemble parts in the physical environment.
NVIDIA Corporation（英伟达公司）；University of Washington（华盛顿大学）
Robotic assembly is one of the oldest and most challenging applications of robotics. In other areas of robotics, such as perception and grasping, simulation has rapidly accelerated research progress, particularly when combined with modern deep learning. However, accurately, efficiently, and robustly simulating the range of contact-rich interactions in assembly remains a longstanding challenge. In this work, we present Factory, a set of physics simulation methods and robot learning tools for such applications. We achieve real-time or faster simulation of a wide range of contact-rich scenes, inc…
We introduce a robotic assembly system that streamlines the design-to-make workflow for going from a CAD model of a product assembly to a fully programmed and adaptive assembly process. Our system captures (in the CAD tool) the intent of the assembly process for a specific robotic workcell and generates a recipe of task-level instructions. By integrating visual sensing with deep-learned perception models, the robots infer the necessary actions to assemble the design from the generated recipe. The perception models are trained directly from simulation, allowing the system to identify various pa…
FZI Research Center for Information Technology（FZI信息技术研究中心）
The programming of robotic assembly tasks is a key component in manufacturing and automation. Force-sensitive assembly, however, often requires reactive strategies to handle slight changes in positioning and unforeseen part jamming. Learning such strategies from human performance is a promising approach, but faces two common challenges: the handling of low part clearances which is difficult to capture from demonstrations and learning intuitive strategies offline without access to the real hardware. We address these two challenges by learning probabilistic force strategies from data that are ea…
In this work, motivated by recent manufacturing trends, we investigate autonomous robotic assembly. Industrial assembly tasks require contact-rich manipulation skills, which are challenging to acquire using classical control and motion planning approaches. Consequently, robot controllers for assembly domains are presently engineered to solve a particular task, and cannot easily handle variations in the product or environment. Reinforcement learning (RL) is a promising approach for autonomously acquiring robot skills that involve contact-rich dynamics. However, RL relies on random exploration f…
ABB Robotics（ABB机器人）；ABB Corporate Research（ABB企业研究院）；Royal Institute of Technology (KTH)（皇家理工学院（KTH））
Industrial robots can solve very complex tasks in controlled environments, but modern applications require robots able to operate in unpredictable surroundings as well. An increasingly popular reactive policy architecture in robotics is Behavior Trees but as with other architectures, programming time still drives cost and limits flexibility. There are two main branches of algorithms to generate policies automatically, automated planning and machine learning, both with their own drawbacks. We propose a method for generating Behavior Trees using a Genetic Programming algorithm and combining the …
Fine robotic assembly, in which the parts to be assembled are small and fragile and lie in an unstructured environment, is still out of reach of today's industrial robots. The main difficulties arise in the precise localization of the parts in an unstructured environment and the control of contact interactions. Our contribution in this paper is twofold. First, we propose a taxonomy of the manipulation primitives that are specifically involved in fine assembly. Such a taxonomy is crucial for designing a scalable robotic system (both hardware and software) given the complexity of real-world asse…
Active Force Control (AFC) is an important scheme for tackling high-precision robotic assembly. Classical force controllers are highly surface-dependent: the controller must be carefully tuned for each type of surface in contact, in order to avoid instabilities and to achieve a reasonable performance level. Here, we build upon the recently-developed Convex Controller Synthesis (CCS) to enable high-precision assembly across a wide range of surface stiffnesses without any surface-dependent tuning. Specifically, we demonstrate peg-in-hole assembly with 100 micron clearance, initial position uncer…
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
Industrial robot manipulators are playing a more significant role in modern manufacturing industries. Though peg-in-hole assembly is a common industrial task which has been extensively researched, safely solving complex high precision assembly in an unstructured environment remains an open problem. Reinforcement Learning (RL) methods have been proven successful in solving manipulation tasks autonomously. However, RL is still not widely adopted on real robotic systems because working with real hardware entails additional challenges, especially when using position-controlled manipulators. The ma…
Tsinghua University（清华大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
In this paper, we present an overview of robotic peg-in-hole assembly and analyze two main strategies: contact model-based and contact model-free strategies. More specifically, we first introduce the contact model control approaches, including contact state recognition and compliant control two steps. Additionally, we focus on a comprehensive analysis of the whole robotic assembly system. Second, without the contact state recognition process, we decompose the contact model-free learning algorithms into two main subfields: learning from demonstrations and learning from environments (mainly base…
Osaka University（大阪大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
This paper develops a robotic manipulation planner for human-robot collaborative assembly. Unlike previous methods which study an independent and fully AI-equipped autonomous system, this paper explores the subtask distribution between a robot and a human and studies a human-in-the-loop robotic system for collaborative assembly. The system distributes the subtasks of an assembly to robots and humans by exploiting their advantages and avoiding their disadvantages. The robot in the system will work on pick-and-place tasks and provide workpieces to humans. The human collaborator will work on fine…
Despite their contributions to the financial efficiency and environmental sustainability of industrial processes, robotic assembly and disassembly have been understudied in the existing literature. This is in contradiction to their importance in realizing the Fourth Industrial Revolution. More specifically, although most of the literature has extensively discussed how to optimally assemble or disassemble given products, the role of other factors has been overlooked. For example, the types of robots involved in implementing the sequence plans, which should ideally be taken into account througho…
Robust task-oriented grasp planning is vital for autonomous robotic precision assembly tasks. Knowledge of the objects' geometry and preconditions of the target task should be incorporated when determining the proper grasp to execute. However, several factors contribute to the challenges of realizing these grasps such as noise when controlling the robot, unknown object properties, and difficulties modeling complex object-object interactions. We propose a method that decomposes this problem and optimizes for grasp robustness, precision, and task performance by learning three cascaded networks. …
Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly
Computer Science and Artificial Intelligence Lab（计算机科学与人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）；Center for Bits and Atoms（比特与原子中心）；Department of Mechanical Engineering（机械工程系）
University of Maryland, College Park（马里兰大学帕克分校）；King Abdullah University of Science & Technology（阿卜杜拉国王科技大学）
We present the design, characterization, and experimental results for a new modular robotic system for programmable self-assembly. The proposed system uses the Hybrid Cube Model (HCM), which integrates classical features from both deterministic and stochastic self-organization models. Thus, for instance, the modules are passive as far as their locomotion is concerned (stochastic), and yet they possess an active undocking routine (deterministic). The robots are constructed entirely from readily accessible components and unlike many existing robots, their excitation is not fluid mediated. Instea…
Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Insertion operations are a critical element of most robotic assembly operation, and peg-in-hole (PiH) insertion is one of the most widely studied tasks in the industrial and academic manipulation communities. PiH insertion is in fact an entire class of problems, where the complexity of the problem can depend on the type of misalignment and contact formation during an insertion attempt. In this paper, we present the design and analysis of adaptive compliance controllers which can be used in insertion-type assembly tasks, including learning-based compliance controllers which can be used for inse…
Bartlett School of Architecture, UCL（伦敦大学学院巴特莱特建筑学院）
Robots-operating autonomous assembly applications in an unstructured environment require precise methods to locate the building components on site. However, the current available object detection systems are not well-optimised for construction applications, due to the tedious setups incorporated for referencing an object to a system and inability to cope with the elements imperfections. In this paper, we propose a flexible object pose estimation framework to enable robots to autonomously handle building components on-site with an error tolerance to build a specific design target without the ne…
Backward Layout Search for Sequence-Constrained Robotic Assembly
College of Computer and Information Sciences, Fujian Agriculture and Forestry University（福建农林大学计算机与信息学院）；School of Mechatronic Engineering and Automation, Shanghai University（上海大学机电工程与自动化学院）；Department of System Innovation, Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院系统创新系）
Autonomous Integration and Improvement of Robotic Assembly using Skill Graph Representations
Department of Electrical and Computer Engineering, Carnegie Mellon University（卡内基梅隆大学电气与计算机工程系）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院）；Chinese University of Hong Kong（香港中文大学）
Existing methods for predicting robotic snap joint assembly cannot predict failures before their occurrence. To address this limitation, this paper proposes a method for predicting error states before the occurence of error, thereby enabling timely recovery. Robotic snap joint assembly requires precise positioning; therefore, even a slight offset between parts can lead to assembly failure. To correctly predict error states, we apply functional principal component analysis (fPCA) to 6D force/torque profiles that are terminated before the occurence of an error. The error state is identified by a…
REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
AssemLM: A Spatial Reasoning Multimodal Large Language Model for Robotic Assembly
Fudan University（复旦大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）；Tianjin University（天津大学）；Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；City University of Hong Kong（香港城市大学）
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology（国立先进工业科学技术研究院）
Robotic assembly tasks involve complex and low-clearance insertion trajectories with varying contact forces at different stages. While the nominal motion trajectory can be easily obtained from human demonstrations through kinesthetic teaching, teleoperation, simulation, among other methods, the force profile is harder to obtain especially when a real robot is unavailable. It is difficult to obtain a realistic force profile in simulation even with physics engines. Such simulated force profiles tend to be unsuitable for the actual robotic assembly due to the reality gap and uncertainty in the as…
WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League
Chair of Machine Learning and Reasoning (i6), RWTH Aachen University（亚琛工业大学机器学习与推理教席（i6））；MASCOR Institute, FH Aachen University of Applied Science（亚琛应用技术大学MASCOR研究所）
While the ultimate goal is to produce a tensegrity more than 6 struts, e.g. a 15-bar tensegrity, past experience has demonstrated that we must first develop an innovative system that will facilitate the assembly of a general n-bar tensegrity. To be successful, we believe the development of the new assembly methodology must encompass not only the design of the clamping system but also the design of the tensegrity itself, including the struts, the springs and the spring-to-strut connectors. We therefore propose to develop the 15-bar in two phases: Phase I will be the development of an innovative…
Robotic force-based compliance control is a preferred approach to achieve high-precision assembly tasks. When the geometric features of assembly objects are asymmetric or irregular, reinforcement learning (RL) agents are gradually incorporated into the compliance controller to adapt to complex force-pose mapping which is hard to model analytically. Since force-pose mapping is strongly dependent on geometric features, a compliance controller is only optimal for current geometric features. To reduce the learning cost of assembly objects with different geometric features, this paper is devoted to…
Seoul National University（首尔国立大学）；Samsung Research（三星研究院）
We propose a novel fast and accurate simulation framework for contact-intensive tight-tolerance robotic assembly tasks. The key components of our framework are as follows: 1) data-driven contact point clustering with a certain variable-input network, which is explicitly trained for simulation accuracy (with real experimental data) and able to accommodate complex/non-convex object shapes; 2) contact force solving, which precisely/robustly enforces physics of contact (i.e., no penetration, Coulomb friction, maximum energy dissipation) with contact mechanics of contact nodes augmented with that o…
X, The Moonshot Factory（X公司，Moonshot工厂）；New England Robotics Validation and Experimentation (NERVE) Center, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校，新英格兰机器人验证与实验中心）
In recent years, many learning based approaches have been studied to realize robotic manipulation and assembly tasks, often including vision and force/tactile feedback. However, it remains frequently unclear what is the baseline state-of-the-art performance and what are the bottleneck problems. In this work, we evaluate some off-the-shelf (OTS) industrial solutions on a recently introduced benchmark, the National Institute of Standards and Technology (NIST) Assembly Task Boards. A set of assembly tasks are introduced and baseline methods are provided to understand their intrinsic difficulty. M…
Massachusetts Institute of Technology（麻省理工学院）；Institute for Systems and Robotics, Instituto Superior Técnico（高等技术学院系统与机器人研究所）
As access to space and robotic autonomy capabilities move forward, there is simultaneously a growing interest in deploying large, complex space structures to provide new on-orbit capabilities. New space-borne observatories, large orbital outposts, and even futuristic on-orbit manufacturing will be enabled by robotic assembly of space structures using techniques like on-orbit additive manufacturing which can provide flexibility in constructing and even repairing complex hardware. However, the dynamics underlying the robotic assembler during manipulation may operate under inertial uncertainties.…
Singapore Centre for 3D Printing (SC3DP)（新加坡3D打印中心）；School of Mechanical and Aerospace Engineering, NTU（南洋理工大学机械与航天工程学院）；Eureka Robotics（尤里卡机器人公司）
This paper explores the idea that skillful assembly is best represented as dynamic sequences of Manipulation Primitives, and that such sequences can be automatically discovered by Reinforcement Learning. Manipulation Primitives, such as "Move down until contact", "Slide along x while maintaining contact with the surface", have enough complexity to keep the search tree shallow, yet are generic enough to generalize across a wide range of assembly tasks. Moreover, the additional "semantics" of the Manipulation Primitives make them more robust in sim2real and against model/environment variations a…
