Multi-robot assembly systems are becoming increasingly appealing in manufacturing due to their ability to automatically, flexibly, and quickly construct desired structural designs. However, effectively planning for these systems in a manner that ensures each robot is simultaneously productive, and not idle, is challenging due to (1) the close proximity that the robots must operate in to manipulate the structure and (2) the inherent structural partial orderings on when each part can be installed. In this paper, we present a task and motion planning framework that jointly plans safe, low-makespan plans for a team of robots to assemble complex spatial structures. Our framework takes a hierarchical approach that, at the high level, uses Mixed-integer Linear Programs to compute an abstract plan comprised of an allocation of robots to tasks subject to precedence constraints and, at the low level, builds on a state-of-the-art algorithm for Multi-Agent Path Finding to plan collision-free robot motions that realize this abstract plan. Critical to our approach is the inclusion of certain collision constraints and movement durations during high-level planning, which better informs the search for abstract plans that are likely to be both feasible and low-makespan while keeping the search tractable. We demonstrate our planning system on several challenging assembly domains with several (sometimes heterogeneous) robots with grippers or suction plates for assembling structures with up to 23 objects involving Lego bricks, bars, plates, or irregularly shaped blocks.
Osaka University（大阪大学）；National Inst. of AIST（国立先进工业科学技术研究所）
Planning dual-arm assembly of more than three objects is a challenging Task and Motion Planning (TAMP) problem. The assembly planner shall consider not only the pose constraints of objects and robots, but also the gravitational constraints that may break the finished part. This paper proposes a planner to plan the dual-arm assembly of more than three objects. It automatically generates the grasp configurations and assembly poses, and simultaneously searches and backtracks the grasp space and assembly space to accelerate the motion planning of robot arms. Meanwhile, the proposed method consider…
Tsinghua University（清华大学）；Beijing Institute of Control Engineering（北京控制工程研究所）；City University of Hong Kong（香港城市大学）
Recent years have seen the emergence of non-cooperative objects in space, like failed satellites and space junk. These objects are usually operated or collected by free-float dual-arm space manipulators. Thanks to eliminating the difficulties of modeling and manual parameter-tuning, reinforcement learning (RL) methods have shown a more promising sign in the trajectory planning of space manipulators. Although previous studies demonstrate their effectiveness, they cannot be applied in tracking dynamic targets with unknown rotation (non-cooperative objects). In this paper, we proposed a learning …
CAMP: Cooperative Arm-Hand Motion Planning in Constrained Spaces
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Great Bay University（大湾区大学）；China Pharmaceutical University（中国药科大学）；KTH Royal Institute of Technology（瑞典皇家理工学院）；The Hong Kong Polytechnic University（香港理工大学）；The University of Hong Kong（香港大学）；University of Belgrade（贝尔格莱德大学）
Beijing Institute of Control Engineering（北京控制工程研究所）；City University of Hong Kong（香港城市大学）；Qian Xuesen Laboratory of Space Technology（钱学森空间技术实验室）；Tsinghua University（清华大学）
Reinforcement learning methods as a promising technique have achieved superior results in the motion planning of free-floating space robots. However, due to the increase in planning dimension and the intensification of system dynamics coupling, the motion planning of dual-arm free-floating space robots remains an open challenge. In particular, the current study cannot handle the task of capturing a non-cooperative object due to the lack of the pose constraint of the end-effectors. To address the problem, we propose a novel algorithm, EfficientLPT, to facilitate RL-based methods to improve plan…
Adaptive Reinforcement and Model Predictive Control Switching for Safe Human-Robot Cooperative Navigation
School of Engineering, The University of Western Australia（西澳大学工程学院）；Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong（香港中文大学机械与自动化工程系）；School of Electrical Engineering and Computer Science, The University of Queensland（昆士兰大学电气工程与计算机科学学院）
Asymmetric Dual-Arm Task Execution using an Extended Relative Jacobian
Division of Robotics, Perception and Learning, KTH Royal Institute of Technology（机器人、感知与学习 division，皇家理工学院）；Dept. of Electrical Eng., Chalmers University of Technology（电气工程系，查尔默斯技术大学）
Tsinghua University（清华大学）；Institut de Rechecherche en Communications et Cybernétique de Nantes（南特通信与网络研究院）
During the conceptual and prototype design stage of an industrial product, it is crucial to take assembly/disassembly and maintenance operations in advance. A well-designed system should enable relatively easy access of operating manipulators in the constrained environment and reduce musculoskeletal disorder risks for those manual handling operations. Trajectory planning comes up as an important issue for those assembly and maintenance operations under a constrained environment, since it determines the accessibility and the other ergonomics issues, such as muscle effort and its related fatigue…
MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion
Faculty of Robot Science and Engineering, Northeastern University（机器人科学与工程学院，东北大学）；Guangdong Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics and School of Intelligence Science and Engineering, the Harbin Institute of Technology Shenzhen（智能变形机制与适应机器人重点实验室和智能科学与工程学院，哈尔滨工业大学深圳）
Design And Control of A Robotic Arm For Industrial Applications
CMR Technical Campus（CMR技术学院）；JNTUH University College of Engineering, Science & Technology Hyderabad（贾瓦哈拉尔·尼赫鲁科技大学海得拉巴工程科学与技术学院）；Holy Mary Institute of Technology（圣玛丽技术学院）
This paper presents a planner that can automatically find an optimal assembly sequence for a dual-arm robot to assemble the soma blocks. The planner uses the mesh model of objects and the final state of the assembly to generate all possible assembly sequence and evaluate the optimal assembly sequence by considering the stability, graspability, assemblability, as well as the need for a second arm. Especially, the need for a second arm is considered when supports from worktables and other workpieces are not enough to produce a stable assembly. The planner will refer to an assisting grasp to addi…
Motion Planning of Cooperative Nonholonomic Mobile Manipulators
Department of Mechanical Engineering, Indian Institute of Technology Bombay（印度理工学院班加罗尔机械工程系）；Center for Systems and Control, Indian Institute of Technology Bombay（印度理工学院班加罗尔系统控制中心）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong（香港中文大学）
In industry assembly lines, parts feeding machines are widely employed as the prologue of the whole procedure. They play the role of sorting the parts randomly placed in bins to the state with specified pose. With the help of the parts feeding machines, the subsequent assembly processes by robot arm can always start from the same condition. Thus it is expected that function of parting feeding machine and the robotic assembly can be integrated with one robot arm. This scheme can provide great flexibility and can also contribute to reduce the cost. The difficulties involved in this scheme lie in…
Robotic manipulators are widely used in modern manufacturing processes. However, their deployment in unstructured environments remains an open problem. To deal with the variety, complexity, and uncertainty of real-world manipulation tasks, it is essential to develop a flexible framework with reduced assumptions on the environment characteristics. In recent years, reinforcement learning (RL) has shown great results for single-arm robotic manipulation. However, research focusing on dual-arm manipulation is still rare. From a classical control perspective, solving such tasks often involves comple…
Synchronized Dual-arm Rearrangement via Cooperative mTSP
National University of Defense Technology（国防科技大学）；Shenzhen University（深圳大学）；Xiangjiang Laboratory（湘江实验室）；Hunan University of Technology and Business（湖南工商大学）
Cyborg Insect Factory: Automatic Assembly System to Build up Insect-computer Hybrid Robot Based on Vision-guided Robotic Arm Manipulation of Custom Bipolar Electrodes
University of Oulu（奥卢大学）；Aalto University（阿尔托大学）；Idiap Research Institute（Idiap研究所）
In this paper we show how different choices regarding compliance affect a dual-arm assembly task. In addition, we present how the compliance parameters can be learned from a human demonstration. Compliant motions can be used in assembly tasks to mitigate pose errors originating from, for example, inaccurate grasping. We present analytical background and accompanying experimental results on how to choose the center of compliance to enhance the convergence region of an alignment task. Then we present the possible ways of choosing the compliant axes for accomplishing alignment in a scenario where…
In this work we propose a learning approach to high-precision robotic assembly problems. We focus on the contact-rich phase, where the assembly pieces are in close contact with each other. Unlike many learning-based approaches that heavily rely on vision or spatial tracking, our approach takes force/torque in task space as the only observation. Our training environment is robotless, as the end-effector is not attached to any specific robot. Trained policies can then be applied to different robotic arms without re-training. This approach can greatly reduce complexity to perform contact-rich rob…
DAG-Plan: Generating Directed Acyclic Dependency Graphs for Dual-Arm Cooperative Planning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences (CASIA)（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院（CASIA））；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学学院）；State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，北京大学计算机科学学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室，OpenGVLab）
VLM-SFD: VLM-Assisted Siamese Flow Diffusion Framework for Dual-Arm Cooperative Manipulation
Department of Computer Science, The University of Manchester（计算机科学系，曼彻斯特大学）；School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学）
PA-BiCoop: A Primary-Auxiliary Cooperative Framework for General Bimanual Manipulation
SGIT AI Lab, State Grid Corporation of China（国家电网公司SGIT人工智能实验室）；The Hong Kong University of Science and Technology, Guangzhou（香港科技大学（广州））；Baidu Research, Beijing, China（百度研究院）；Zhejiang University of Technology, Hangzhou（浙江工业大学）
A vacuum lifter is widely used to hold and pick up large, heavy, and flat objects. Conventionally, when using a vacuum lifter, a human worker watches the state of a running vacuum lifter and adjusts the object's pose to maintain balance. In this work, we propose using a dual-arm robot to replace the human workers and develop planning and control methods for a dual-arm robot to raise a heavy plate with the help of a vacuum lifter. The methods help the robot determine its actions by considering the vacuum lifer's suction position and suction force limits. The essence of the methods is two-fold. …
Google Research（谷歌研究院）；Carnegie Mellon University（卡内基梅隆大学）
Most successes in robotic manipulation have been restricted to single-arm gripper robots, whose low dexterity limits the range of solvable tasks to pick-and-place, inser-tion, and object rearrangement. More complex tasks such as assembly require dual and multi-arm platforms, but entail a suite of unique challenges such as bi-arm coordination and collision avoidance, robust grasping, and long-horizon planning. In this work we investigate the feasibility of training deep reinforcement learning (RL) policies in simulation and transferring them to the real world (Sim2Real) as a generic methodology…
IBM Research - Tokyo, IBM Japan（IBM日本东京研究院）；Tsukuba Research Laboratory, Yaskawa Electric Corporation（安川电机筑波研究所）
High precision assembly of mechanical parts requires accuracy exceeding the robot precision. Conventional part mating methods used in the current manufacturing requires tedious tuning of numerous parameters before deployment. We show how the robot can successfully perform a tight clearance peg-in-hole task through training a recurrent neural network with reinforcement learning. In addition to saving the manual effort, the proposed technique also shows robustness against position and angle errors for the peg-in-hole task. The neural network learns to take the optimal action by observing the rob…
Italian Institute of Technology（意大利技术研究院）；Politecnico di Milano（米兰理工大学）；Xi’an Jiaotong University（西安交通大学）；Hanyang University ERICA（汉阳大学ERICA校区）
This article proposes a method for learning and robotic replication of dynamic collaborative tasks from offline videos. The objective is to extend the concept of learning from demonstration (LfD) to dynamic scenarios, benefiting from widely available or easily producible offline videos. To achieve this goal, we decode important dynamic information, such as the Configuration Dependent Stiffness (CDS), which reveals the contribution of arm pose to the arm endpoint stiffness, from a three-dimensional human skeleton model. Next, through encoding of the CDS via Gaussian Mixture Model (GMM) and deco…
Osaka University（大阪大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
This paper develops a robotic manipulation planner for human-robot collaborative assembly. Unlike previous methods which study an independent and fully AI-equipped autonomous system, this paper explores the subtask distribution between a robot and a human and studies a human-in-the-loop robotic system for collaborative assembly. The system distributes the subtasks of an assembly to robots and humans by exploiting their advantages and avoiding their disadvantages. The robot in the system will work on pick-and-place tasks and provide workpieces to humans. The human collaborator will work on fine…
A Coordinated Dual-Arm Framework for Delicate Snap-Fit Assemblies
IITGN Robotics Laboratory, Department of Mechanical Engineering, Indian Institute of Technology Gandhinagar（印度理工学院冈丁加尔学院机器人实验室，机械工程系）；Addverb Technologies Pvt. Ltd.（Addverb科技私人有限公司）；Addverb Technologies Limited（Addverb科技有限公司）
Osaka University（大阪大学）；National Inst. of AIST（国立产业技术综合研究所）
This paper presents a manipulation planning algorithm for robots to reorient objects. It automatically finds a sequence of robot motion that manipulates and prepares an object for specific tasks. Examples of the preparatory manipulation planning problems include reorienting an electric drill to cut holes, reorienting workpieces for assembly, and reorienting cargo for packing, etc. The proposed algorithm could plan single and dual arm manipulation sequences to solve the problems. The mechanism under the planner is a regrasp graph which encodes grasp configurations and object poses. The algorith…
The Brain-Inspired Cooperative Shared Control Framework for Brain-Machine Interface
Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Tsinghua University（清华大学）
Standardized evaluation measures have aided in the progress of machine learning approaches in disciplines such as computer vision and machine translation. In this paper, we make the case that robotic learning would also benefit from benchmarking, and present the "REPLAB" platform for benchmarking vision-based manipulation tasks. REPLAB is a reproducible and self-contained hardware stack (robot arm, camera, and workspace) that costs about 2000 USD, occupies a cuboid of size 70x40x60 cm, and permits full assembly within a few hours. Through this low-cost, compact design, REPLAB aims to drive wid…
Many high precision (dis)assembly tasks are still being performed by humans, whereas this is an ideal opportunity for automation. This paper provides a framework which enables a non-expert human operator to teach a robotic arm to do complex precision tasks. The framework uses a variable Cartesian impedance controller to execute trajectories learned from kinesthetic human demonstrations. Feedback can be given to interactively reshape or speed up the original demonstration. Board localization is done through a visual estimation of the task board position and refined through haptic feedback. Our …
Osaka University（大阪大学）；National Institute of Advanced Industrial Science and Technology（产业技术综合研究所）
This paper proposes a robot assembly planning method by automatically reading the graphical instruction manuals design for humans. Essentially, the method generates an Assembly Task Sequence Graph (ATSG) by recognizing a graphical instruction manual. An ATSG is a graph describing the assembly task procedure by detecting types of parts included in the instruction images, completing the missing information automatically, and correcting the detection errors automatically. To build an ATSG, the proposed method first extracts the information of the parts contained in each image of the graphical ins…
University of Colorado Boulder（科罗拉多大学博尔德分校）；Vrije Universiteit Brussel（布鲁塞尔自由大学）；Flanders Make（佛兰德斯制造）
We consider solving a cooperative multi-robot object manipulation task using reinforcement learning (RL). We propose two distributed multi-agent RL approaches: distributed approximate RL (DA-RL), where each agent applies Q-learning with individual reward functions; and game-theoretic RL (GT-RL), where the agents update their Q-values based on the Nash equilibrium of a bimatrix Q-value game. We validate the proposed approaches in the setting of cooperative object manipulation with two simulated robot arms. Although we focus on a small system of two agents in this paper, both DA-RL and GT-RL app…
Efficient and robust task planning for a human-robot collaboration (HRC) system remains challenging. The human-aware task planner needs to assign jobs to both robots and human workers so that they can work collaboratively to achieve better time efficiency. However, the complexity of the tasks and the stochastic nature of the human collaborators bring challenges to such task planning. To reduce the complexity of the planning problem, we utilize the hierarchical task model, which explicitly captures the sequential and parallel relationships of the task. We model human movements with the sigma-lo…
