We consider solving a cooperative multi-robot object manipulation task using reinforcement learning (RL). We propose two distributed multi-agent RL approaches: distributed approximate RL (DA-RL), where each agent applies Q-learning with individual reward functions; and game-theoretic RL (GT-RL), where the agents update their Q-values based on the Nash equilibrium of a bimatrix Q-value game. We validate the proposed approaches in the setting of cooperative object manipulation with two simulated robot arms. Although we focus on a small system of two agents in this paper, both DA-RL and GT-RL apply to general multi-agent systems, and are expected to scale well to large systems.
Multi-robot cooperation requires agents to make decisions that are consistent with the shared goal without disregarding action-specific preferences that might arise from asymmetry in capabilities and individual objectives. To accomplish this goal, we propose a method named SLiCC: Stackelberg Learning in Cooperative Control. SLiCC models the problem as a partially observable stochastic game composed of Stackelberg bimatrix games, and uses deep reinforcement learning to obtain the payoff matrices associated with these games. Appropriate cooperative actions are then selected with the derived Stac…
Volkswagen Group（大众集团）；Machine Learning Research Lab（机器学习研究实验室）
Multi-robot manipulation tasks involve various control entities that can be separated into dynamically independent parts. A typical example of such real-world tasks is dual-arm manipulation. Learning to naively solve such tasks with reinforcement learning is often unfeasible due to the sample complexity and exploration requirements growing with the dimensionality of the action and state spaces. Instead, we would like to handle such environments as multi-agent systems and have several agents control parts of the whole. However, decentralizing the generation of actions requires coordination acro…
At present, the research on robot team cooperation is still in qualitative analysis phase and lacks the description model that can quantitatively describe the dynamical evolution of team cooperative relationships with constantly changeable task demand in Multi-robot field. First this paper whole and static describes organization model HWROM of robot team, then uses Markov course and Bayesian theorem for reference, dynamical describes the team cooperative relationships building. Finally from cooperative entity layer, ability layer and relative layer we research team formation and cooperative me…
University of Calabria（卡拉布里亚大学）；Middlesex University London（伦敦密德萨斯大学）
The mine detection in an unexplored area is an optimization problem where multiple mines, randomly distributed throughout an area, need to be discovered and disarmed in a minimum amount of time. We propose a strategy to explore an unknown area, using a stigmergy approach based on ants behavior, and a novel swarm based protocol to recruit and coordinate robots for disarming the mines cooperatively. Simulation tests are presented to show the effectiveness of our proposed Ant-based Task Robot Coordination (ATRC) with only the exploration task and with both exploration and recruiting strategies. M…
University of Zagreb（萨格勒布大学）；Faculty of Electrical Engineering and Computing（电气工程与计算学院）
In this paper, we study a cooperative aerial-ground robotic team and its application to the task of automated construction. We propose a solution for planning and coordinating the mission of constructing a wall with a predefined structure for a heterogeneous system consisting of one mobile robot and up to three unmanned aerial vehicles. The wall consists of bricks of various weights and sizes, some of which need to be transported using multiple robots simultaneously. To that end, we use hierarchical task representation to specify interrelationships between mission subtasks and employ effective…
Underlying relationships among multiagent systems (MAS) in hazardous scenarios can be represented as game-theoretic models. In adversarial environments, the adversaries can be intentional or unintentional based on their needs and motivations. Agents will adopt suitable decision-making strategies to maximize their current needs and minimize their expected costs. This paper proposes and extends the new hierarchical network-based model, termed Game-theoretic Utility Tree (GUT), to arrive at a cooperative pursuit strategy to catch an evader in the Pursuit-Evasion game domain. We verify and demonst…
Safe and efficient co-planning of multiple robots in pedestrian participation environments is promising for applications. In this work, a novel multi-robot social-aware efficient cooperative planner that on the basis of off-policy multi-agent reinforcement learning (MARL) under partial dimension-varying observation and imperfect perception conditions is proposed. We adopt temporal-spatial graph (TSG)-based social encoder to better extract the importance of social relation between each robot and the pedestrians in its field of view (FOV). Also, we introduce K-step lookahead reward setting in mu…
In this paper, we introduce ChoiRbot, a toolbox for distributed cooperative robotics based on the novel Robot Operating System (ROS) 2. ChoiRbot provides a fully-functional toolset to execute complex distributed multi-robot tasks, either in simulation or experimentally, with a particular focus on networks of heterogeneous robots without a central coordinator. Thanks to its modular structure, ChoiRbot allows for a highly straight implementation of optimization-based distributed control schemes, such as distributed optimal control, model predictive control, task assignment, in which local comput…
Southern University of Science and Technology（南方科技大学）
Multi-robot localization has been a critical problem for robots performing complex tasks cooperatively. In this paper, we propose a decentralized approach to localize a group of robots in a large featureless environment. The proposed approach only requires that at least one robot remains stationary as a temporary landmark during a certain period of time. The novelty of our approach is threefold: (1) developing a decentralized scheme that each robot calculates their own state and only stores the latest one to reduce storage and computational cost, (2) developing an efficient localization algori…
It is of great challenge, though promising, to coordinate collective robots for hunting an evader in a decentralized manner purely in light of local observations. In this paper, this challenge is addressed by a novel hybrid cooperative pursuit algorithm that combines reinforcement learning with the artificial potential field method. In the proposed algorithm, decentralized deep reinforcement learning is employed to learn cooperative pursuit policies that are adaptive to dynamic environments. The artificial potential field method is integrated into the learning process as predefined rules to im…
The Hong Kong Polytechnic University（香港理工大学）；Shanghai University（上海大学）
As a disruptive technology, blockchain, particularly its original form of bitcoin as a type of digital currency, has attracted great attentions. The innovative distributed decision making and security mechanism lay the technical foundation for its success, making us consider to penetrate the power of blockchain technology to distributed control and cooperative robotics, in which the distributed and secure mechanism is also highly demanded. Actually, security and distributed communication have long been unsolved problems in the field of distributed control and cooperative robotics. It has been …
Technical University of Darmstadt（达姆施塔特工业大学）；Mitsubishi Electric Research Laboratories（三菱电机研究实验室）
Robotic manipulation stands as a largely unsolved problem despite significant advances in robotics and machine learning in recent years. One of the key challenges in manipulation is the exploration of the dynamics of the environment when there is continuous contact between the objects being manipulated. This paper proposes a model-based active exploration approach that enables efficient learning in sparse-reward robotic manipulation tasks. The proposed method estimates an information gain objective using an ensemble of probabilistic models and deploys model predictive control (MPC) to plan act…
Princeton University（普林斯顿大学）；University of Bergamo（贝加莫大学）；Skidmore, Owings & Merrill (SOM)（斯基德莫尔、奥因斯与梅里尔建筑事务所（SOM））
Geometrically complex masonry structures (e.g., arches, domes, vaults) are traditionally built with expensive scaffolding or falsework to provide stability during construction. The process of building such structures can potentially be improved through the use of multiple robots working together in a cooperative assembly framework. Here a robot is envisioned as both a placement and external support agent during fabrication - the unfinished structure is supported in such a way that scaffolding is not required. The goal of this paper is to present and validate the efficacy of three cooperative f…
Research in multi-robot and swarm systems has seen significant interest in cooperation of agents in complex and dynamic environments. To effectively adapt to unknown environments and maximize the utility of the group, robots need to cooperate, share information, and make a suitable plan according to the specific scenario. Inspired by Maslow's hierarchy of human needs and systems theory, we introduce Robot's Need Hierarchy and propose a new solution called Self-Adaptive Swarm System (SASS). It combines multi-robot perception, communication, planning, and execution with the cooperative managemen…
Task-priority Intermediated Hierarchical Distributed Policies: Reinforcement Learning of Adaptive Multi-robot Cooperative Transport
Graduate School of Information Science, Nara Institute of Science and Technology (NAIST)（奈良科学技术大学信息科学研究生院）；R-Frontier Division, Frontier Research Center, Toyota Motor Corporation（丰田汽车公司前沿研究中心R前沿部）
In this paper we address the challenge of exploration in deep reinforcement learning for robotic manipulation tasks. In sparse goal settings, an agent does not receive any positive feedback until randomly achieving the goal, which becomes infeasible for longer control sequences. Inspired by touch-based exploration observed in children, we formulate an intrinsic reward based on the sum of forces between a robot's force sensors and manipulation objects that encourages physical interaction. Furthermore, we introduce contact-prioritized experience replay, a sampling scheme that prioritizes contact…
Deep Reinforcement Learning (DRL) has emerged as a powerful control technique in robotic science. In contrast to control theory, DRL is more robust in the thorough exploration of the environment. This capability of DRL generates more human-like behaviour and intelligence when applied to the robots. To explore this capability, we designed challenging manipulation tasks to observe robots strategy to handle complex scenarios. We observed that robots not only perform tasks successfully, but also transpire a creative and non intuitive solution. We also observed robot's persistence in tasks that are…
University of California, Riverside（加州大学河滨分校）；University of Delaware（特拉华大学）
In this paper we present a consistent and distributed state estimator for multi-robot cooperative localization (CL) which efficiently fuses environmental features and loop-closure constraints across time and robots. In particular, we leverage covariance intersection (CI) to allow each robot to only estimate its own state and autocovariance and compensate for the unknown correlations between robots. Two novel multi-robot methods for utilizing common environmental SLAM features are introduced and evaluated in terms of accuracy and efficiency. Moreover, we adapt CI to enable drift-free estimation…
Research on multi-robot systems has demonstrated promising results in manifold applications and domains. Still, efficiently learning an effective robot behaviors is very difficult, due to unstructured scenarios, high uncertainties, and large state dimensionality (e.g. hyper-redundant and groups of robot). To alleviate this problem, we present Q-CP a cooperative model-based reinforcement learning algorithm, which exploits action values to both (1) guide the exploration of the state space and (2) generate effective policies. Specifically, we exploit Q-learning to attack the curse-of-dimensionali…
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；Research Center for Information Technology (FZI)（信息技术研究中心）
In order to fully exploit the advantages inherent to cooperating heterogeneous multi-robot teams, sophisticated coordination algorithms are essential. Time-extended multi-robot task allocation approaches assign and schedule a set of tasks to a group of robots such that certain objectives are optimized and operational constraints are met. This is particularly challenging if cooperative tasks, i.e. tasks that require two or more robots to work directly together, are considered. In this paper, we present an easy-to-implement criterion to validate the feasibility, i.e. executability, of solutions …
Toyota Motor Corporation（丰田汽车公司）；Nara Institute of Science and Technology（奈良科学技术研究所）
We consider task allocation for multi-object transport using a multi-robot system, in which each robot selects one object among multiple objects with different and unknown weights. The existing centralized methods assume the number of robots and tasks to be fixed, which is inapplicable to scenarios that differ from the learning environment. Meanwhile, the existing distributed methods limit the minimum number of robots and tasks to a constant value, making them applicable to various numbers of robots and tasks. However, they cannot transport an object whose weight exceeds the load capacity of r…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Multi-agent reinforcement learning (MARL) has achieved remarkable success in various challenging problems. Meanwhile, more and more benchmarks have emerged and provided some standards to evaluate the algorithms in different fields. On the one hand, the virtual MARL environments lack knowledge of real-world tasks and actuator abilities, and on the other hand, the current task-specified multi-robot platform has poor support for the generality of multi-agent reinforcement learning algorithms and lacks support for transferring from simulation to the real environment. Bridging the gap between the v…
Consistent Distributed Cooperative Localization: A Coordinate Transformation Approach
National Key Laboratory of Modeling and Simulation for Complex Systems, School of Astronautics, Harbin Institute of Technology（哈尔滨工业大学航天学院复杂系统建模与仿真国家重点实验室）
Nav-SCOPE: Swarm Robot Cooperative Perception and Coordinated Navigation
Department of Automation, Tsinghua University（自动化系，清华大学）；Beijing National Research Center for Information Science and Technology, Tsinghua University（北京信息科学与技术国家研究中心，清华大学）；School of Mechanical and Electrical Engineering, Beijing Information Science and Technology University（机械与电子工程学院，北京信息科学与技术大学）
We present a self-improving, Neural Tree Expansion (NTE) method for multi-robot online planning in non-cooperative environments, where each robot attempts to maximize its cumulative reward while interacting with other self-interested robots. Our algorithm adapts the centralized, perfect information, discrete-action space method from AlphaZero to a decentralized, partial information, continuous action space setting for multi-robot applications. Our method has three interacting components: (i) a centralized, perfect-information "expert" Monte Carlo Tree Search (MCTS) with large computation resou…
CoLF: Learning Consistent Leader-Follower Policies for Vision-Language-Guided Multi-Robot Cooperative Transport
Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology (NAIST)（信息科学系，科学技术研究生学校，科学与技术国立研究所（NAIST））；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院（EPFL））
University of Cincinnati（辛辛那提大学）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）
Object transportation could be a challenging problem for a single robot due to the oversize and/or overweight issues. A multi-robot system can take the advantage of increased driving power and more flexible configuration to solve such a problem. However, increased number of individuals also changed the dynamics of the system which makes control of a multi-robot system more complicated. Even worse, if the whole system is sitting on a centralized decision making unit, the data flow could be easily overloaded due to the upscaling of the system. In this research, we propose a decentralized control…
University of Tabriz（大不里士大学）；Sharif University of Technology（谢里夫理工大学）；Middle East Technical University（中东技术大学）；Czech Technical University（捷克技术大学）；The University of Manchester（曼彻斯特大学）
Using robots for exploration of extreme and hazardous environments has the potential to significantly improve human safety. For example, robotic solutions can be deployed to find the source of a chemical leakage and clean the contaminated area. This paper demonstrates a proof-of-concept bio-inspired exploration method using a swarm robotic system, which is based on a combination of two bio-inspired behaviours: aggregation, and pheromone tracking. The main idea of the work presented is to follow pheromone trails to find the source of a chemical leakage and then carry out a decontamination task …
Massachusetts Institute of Technology（麻省理工学院）；Woods Hole Oceanographic Institution（伍兹霍尔海洋研究所）
We present a novel path-planning algorithm to reduce localization error for a network of robots cooperatively localizing via inter-robot range measurements. The quality of localization with range measurements depends on the configuration of the network, and poor configurations can cause substantial localization errors. To reduce the effect of network configuration on localization error for moving networks we consider various optimality measures of the Fisher information matrix (FIM), which have well-studied relationships with the localization error. In particular, we pose a trajectory planning…
California State Polytechnic University, Pomona（加州州立理工大学波莫纳分校）；Jet Propulsion Laboratory, California Institute of Technology（加州理工学院喷气推进实验室）；Brunel University London（伦敦布鲁内尔大学）
The intersection of Quantum Technologies and Robotics Autonomy is explored in the present paper. The two areas are brought together in establishing an interdisciplinary interface that contributes to advancing the field of system autonomy, and pushes the engineering boundaries beyond the existing techniques. The present research adopts the experimental aspects of quantum entanglement and quantum cryptography, and integrates these established quantum capabilities into distributed robotic platforms, to explore the possibility of achieving increased autonomy for the control of multi-agent robotic …
The present work provides a new approach to solve the well-known multi-robot co-operative box pushing problem as a multi objective optimization problem using modified Multi-objective Particle Swarm Optimization. The method proposed here allows both turning and translation of the box, during shift to a desired goal position. We have employed local planning scheme to determine the magnitude of the forces applied by the two mobile robots perpendicularly at specific locations on the box to align and translate it in each distinct step of motion of the box, for minimization of both time and energy. …
For tasks where the dynamics of multiple agents are physically coupled, e.g., in cooperative manipulation, the coordination between the individual agents becomes crucial, which requires exact knowledge of the interaction dynamics. This problem is typically addressed using centralized estimators, which can negatively impact the flexibility and robustness of the overall system. To overcome this shortcoming, we propose a novel distributed learning framework for the exemplary task of cooperative manipulation using Bayesian principles. Using only local state information each agent obtains an estima…
This paper addresses the problem of cooperative manipulation of a single object by N robotic agents under local goal specifications given as Metric Interval Temporal Logic (MITL) formulas. In particular, we propose a distributed model-free control protocol for the trajectory tracking of the cooperatively manipulated object without necessitating feedback of the contact forces/torques or inter-agent communication. This allows us to abstract the motion of the coupled object-agents system as a finite transition system and, by employing standard automata-based methodologies, we derive a hybrid cont…
A cooperative positioning flamework of human and robots based on visible light communication (VLC) is proposed. Based on the experiment system, we demonstrated it is feasible and has high-accuracy and real-time performance.
This paper describes an approach to the design of a population of cooperative robots based on concepts borrowed from Systems Theory and Artificial Intelligence. The research has been developed under the SocRob project, carried out by the Intelligent Systems Laboratory at the Institute for Systems and Robotics - Instituto Superior Tecnico (ISR/IST) in Lisbon. The acronym of the project stands both for "Society of Robots" and "Soccer Robots", the case study where we are testing our population of robots. Designing soccer robots is a very challenging problem, where the robots must act not only to …
SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）；College of Artificial Intelligence, Nanjing Forestry University（南京林业大学人工智能学院）
