We present algorithms and results for a robotic manipulation system that was designed to be easily programmable and adaptable to various tasks common to industrial setting, which is inspired by the Industrial Assembly Challenge at the 2018 World Robotics Summit in Tokyo. This challenge included assembly of standard, commercially available industrial parts into 2D and 3D assemblies. We demonstrate three tasks that can be classified into "peg-in-hole" and "hole-on-peg" tasks and identify two canonical algorithms: spiral-based search and tilting insertion. Both algorithms use hand-coded thresholds in the force and torque domains to detect critical points in the assembly. After briefly summarizing the state of the art in research, we describe the strategy and approach utilized by the tested system, how it's design bears on its performance, statistics on 20 experimental trials for each task, lessons learned during the development of the system, and open research challenges that still remain.
REASSEMBLE: A Multimodal Dataset for Contact-rich Robotic Assembly and Disassembly
Autonomous Systems Lab, Institute of Computer Technology, TU Wien（自主系统实验室，计算机技术学院，维也纳技术大学）；Institute of Robotics and Mechatronics (DLR), German Aerospace Center（机器人与机电系统研究所（DLR），德国航空航天中心）
Beihang University（北京航空航天大学）；Beijing Institute of Aeronautical Materials（北京航空材料研究院）
Object dropping may occur when the robotic arm grasps objects with uneven mass distribution due to additional moments generated by objects' gravity. To solve this problem, we present a novel work that does not require extra wrist and tactile sensors and large amounts of experiments for learning. First, we obtain the center-of-mass position of the rod object using the widely fixed joint torque sensors on the robot arm and RGBD camera. Further, we give the strategy of grasping to improve grasp stability. Simulation experiments are performed in "Mujoco". Results demonstrate that our work is effec…
Vision-based robotic assembly is a crucial yet challenging task as the interaction with multiple objects requires high levels of precision. In this paper, we propose an integrated 6D robotic system to perceive, grasp, manipulate and assemble blocks with tight tolerances. Aiming to provide an off-the-shelf RGB-only solution, our system is built upon a monocular 6D object pose estimation network trained solely with synthetic images leveraging physically-based rendering. Subsequently, pose-guided 6D transformation along with collision-free assembly is proposed to construct any designed structure …
Fine Robotic Manipulation without Force/Torque Sensor
School of Mechanical and Aerospace Engineering, Nanyang Technological University（南洋理工大学机械与宇航工程学院）；Eureka Robotics（尤里卡机器人公司）；Singapore Centre for 3D Printing (SC3DP)（新加坡3D打印中心）
University of California, Berkeley（加州大学伯克利分校）；Siemens Corporate Technology（西门子公司技术部）；Technion（以色列理工学院）
Precise robotic manipulation skills are desirable in many industrial settings, reinforcement learning (RL) methods hold the promise of acquiring these skills autonomously. In this paper, we explicitly consider incorporating operational space force/torque information into reinforcement learning; this is motivated by humans heuristically mapping perceived forces to control actions, which results in completing high-precision tasks in a fairly easy manner. Our approach combines RL with force/torque information by incorporating a proper operational space force controller; where we also exploit diff…
Fraunhofer Institute for Manufacturing Engineering and Automation（弗劳恩霍夫制造工程与自动化研究所）；Technical University of Munich（慕尼黑工业大学）
Gear assembly is an essential but challenging task in industrial automation. This paper presents a novel two-stage approach for achieving high-precision and flexible gear assembly. The proposed approach integrates YOLO to coarsely localize the workpiece in a searching phase and deep reinforcement learning (DRL) to complete the insertion. Specifically, DRL addresses the challenge of partial visibility when the on-wrist camera is too close to the workpiece. Additionally, force feedback is used to smoothly transit the process from the first phase to the second phase. To reduce the data collection…
Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院）；Chinese University of Hong Kong（香港中文大学）
Existing methods for predicting robotic snap joint assembly cannot predict failures before their occurrence. To address this limitation, this paper proposes a method for predicting error states before the occurence of error, thereby enabling timely recovery. Robotic snap joint assembly requires precise positioning; therefore, even a slight offset between parts can lead to assembly failure. To correctly predict error states, we apply functional principal component analysis (fPCA) to 6D force/torque profiles that are terminated before the occurence of an error. The error state is identified by a…
Precision robotic manipulation tasks (insertion, screwing, precisely pick, precisely place) are required in many scenarios. Previous methods achieved good performance on such manipulation tasks. However, such methods typically require tedious calibration or expensive sensors. 3D/RGB-D cameras and torque/force sensors add to the cost of the robotic application and may not always be economical. In this work, we aim to solve these but using only weak-calibrated and low-cost webcams. We propose Binocular Alignment Learning (BAL), which could automatically learn the eye-hand coordination and points…
The development of a generic and effective force model for semi-automatic or manual virtual assembly with haptic support is not a trivial task, especially when the assembly constraints involve complex features of arbitrary shape. The primary challenge lies in a proper formulation of the guidance forces and torques that effectively assist the user in the exploration of the virtual environment (VE), from repulsing collisions to attracting proper contact. The secondary difficulty is that of efficient implementation that maintains the standard 1 kHz haptic refresh rate. We propose a purely geometr…
ManipForce: Force-Guided Policy Learning with Frequency-Aware Representation for Contact-Rich Manipulation
Department of AI Convergence, Gwangju Institute of Science and Technology (GIST)（人工智能融合系，全州科学技术院）；Department of AI Machinery, Korea Institute of Machinery & Materials (KIMM)（人工智能机械系，韩国机械材料研究院）
Mastering Autonomous Assembly in Fusion Application with Learning-by-doing: a Peg-in-hole Study
Lappeenranta University of Technology（拉彭兰塔工业大学）；Institute of Plasma Physics Chinese Academy of Sciences（中国科学院等离子体物理研究所）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）
FZI Research Center for Information Technology（FZI信息科技研究中心）；IAR Institute for Anthropomatics and Robotics, HIS Humanoids and Intelligence Systems Lab, KIT Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院（KIT）IAR人形机器人与机器人学研究所、HIS人形与智能系统实验室）
Robotic automation is a key driver for the advancement of technology. The skills of human workers, however, are difficult to program and seem currently unmatched by technical systems. In this work we present a data-driven approach to extract and learn robot-independent contact skills from human demonstrations in simulation environments, using a Long Short Term Memory (LSTM) network. Our model learns to generate error-correcting sequences of forces and torques in task space from object-relative motion, which industrial robots carry out through a Cartesian force control scheme on the real setup.…
Context-Based Meta Reinforcement Learning for Robust and Adaptable Peg-in-Hole Assembly Tasks
Humanoid Robots Lab and Center for Robotics, University of Bonn, Germany（波恩大学人形机器人实验室和机器人中心）；Lamarr Institute for Machine Learning and Artificial Intelligence, Bonn, Germany（波恩大学拉马尔人工智能与机器学习研究所）；Cyber Physical Systems Lab, Egypt Japan University of Science and Technology, Alexandria, Egypt（埃及日本科学技术大学网络物理系统实验室）；Faculty of Engineering, Alexandria University, Alexandria, Egypt（亚历山大大学工程学院）；Mechatronics Department, Ain Shams University, Cairo, Egypt（亚历山大大学机电系）
From Reach to Insert: Tactile-Augmented Precision Assembly under Sub-Millimeter Tolerances
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
In this work we propose a learning approach to high-precision robotic assembly problems. We focus on the contact-rich phase, where the assembly pieces are in close contact with each other. Unlike many learning-based approaches that heavily rely on vision or spatial tracking, our approach takes force/torque in task space as the only observation. Our training environment is robotless, as the end-effector is not attached to any specific robot. Trained policies can then be applied to different robotic arms without re-training. This approach can greatly reduce complexity to perform contact-rich rob…
Enhancing Autonomous Manipulator Control with Human-in-loop for Uncertain Assembly Environments
Space Robotics Lab. (SRL) in Department of Aerospace Engineering, Graduate School of Engineering, Tohoku University（太空机器人实验室（SRL）位于航空航天工程系，工学研究生院，东北大学）
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
We present the design of a learning-based compliance controller for assembly operations for industrial robots. We propose a solution within the general setting of learning from demonstration (LfD), where a nominal trajectory is provided through demonstration by an expert teacher. This can be used to learn a suitable representation of the skill that can be generalized to novel positions of one of the parts involved in the assembly, for example the hole in a peg-in-hole (PiH) insertion task. Under the expectation that this novel position might not be entirely accurately estimated by a vision or …
Osaka University（大阪大学）；National Inst. of AIST（国立先进工业科学技术研究所）
This paper uses robots to assemble pegs into holes on surfaces with different colors and textures. It especially targets at the problem of peg-in-hole assembly with initial position uncertainty. Two in-hand cameras and a force-torque sensor are used to account for the position uncertainty. A program sequence comprising learning-based visual servoing, spiral search, and impedance control is implemented to perform the peg-in-hole task with feedback from the above sensors. Contributions are mainly made in the learning-based visual servoing of the sequence, where a deep neural network is trained w…
Uncertainty in perception, actuation, and the environment often require multiple attempts for a robotic task to be successful. We study a class of problems providing (1) low-entropy indicators of terminal success / failure, and (2) unreliable (high-entropy) data to predict the final outcome of an ongoing task. Examples include a robot trying to connect with a charging station, parallel parking, or assembling a tightly-fitting part. The ability to restart after predicting failure early, versus simply running to failure, can significantly decrease the makespan, that is, the total time to complet…
PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院通用人工智能国家重点实验室）；Delta Intelligence（三角洲智能公司）；PKU-Wuhan Institute for Artificial Intelligence（北京大学武汉人工智能研究院）；Hubei Humanoid Robot Innovation Center Co., Ltd.（湖北人形机器人创新中心有限公司）；China Academy of Information and Communications Technology（中国信息通信研究院）
Cornell University（康奈尔大学）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
One of the main challenges in peg-in-a-hole (PiH) insertion tasks is in handling the uncertainty in the location of the target hole. In order to address it, high-dimensional sensor inputs from sensor modalities such as vision, force/torque sensing, and proprioception can be combined to learn control policies that are robust to this uncertainty in the target pose. Whereas deep learning has shown success in recognizing objects and making decisions with high-dimensional inputs, the learning procedure might damage the robot when applying directly trial- and-error algorithms on the real system. At …
ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing
School of Informatics, University of Edinburgh（信息学院，爱丁堡大学）；Department of Mechanical and Mechatronics Engineering, University of Waterloo（机械与机电工程系，滑铁卢大学）
University of Applied Sciences Augsburg（奥格斯堡应用技术大学）；Ulster University（阿尔斯特大学）
A robust grip is key to successful manipulation and joining of work pieces involved in any industrial assembly process. Stability of a grip depends on geometric and physical properties of the object as well as the gripper itself. Current state-of-the-art algorithms can usually predict if a grip would fail. However, they are not able to predict the force at which the gripped object starts to slip, which is critical as the object might be subjected to external forces, e.g. when joining it with another object. This research project aims to develop a AI-based approach for a grip metric based on ta…
