Traditional radio systems are strictly co-designed on the lower levels of the OSI stack for compatibility and efficiency. Although this has enabled the success of radio communications, it has also introduced lengthy standardization processes and imposed static allocation of the radio spectrum. Various initiatives have been undertaken by the research community to tackle the problem of artificial spectrum scarcity by both making frequency allocation more dynamic and building flexible radios to replace the static ones. There is reason to believe that just as computer vision and control have been overhauled by the introduction of machine learning, wireless communication can also be improved by utilizing similar techniques to increase the flexibility of wireless networks. In this work, we pose the problem of discovering low-level wireless communication schemes ex-nihilo between two agents in a fully decentralized fashion as a reinforcement learning problem. Our proposed approach uses policy gradients to learn an optimal bi-directional communication scheme and shows surprisingly sophisticated and intelligent learning behavior. We present the results of extensive experiments and an analysis of the fidelity of our approach.
Deep reinforcement learning for multi-agent cooperation and competition has been a hot topic recently. This paper focuses on cooperative multi-agent problem based on actor-critic methods under local observations settings. Multi agent deep deterministic policy gradient obtained state of art results for some multi-agent games, whereas, it cannot scale well with growing amount of agents. In order to boost scalability, we propose a parameter sharing deterministic policy gradient method with three variants based on neural networks, including actor-critic sharing, actor sharing and actor sharing wit…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Almost all multi-agent reinforcement learning algorithms without communication follow the principle of centralized training with decentralized execution. During centralized training, agents can be guided by the same signals, such as the global state. During decentralized execution, however, agents lack the shared signal. Inspired by viewpoint invariance and contrastive learning, we propose consensus learning for cooperative multi-agent reinforcement learning in this paper. Although based on local observations, different agents can infer the same consensus in discrete space. During decentralize…
Alexa AI, Amazon Inc（亚马逊公司Alexa人工智能部门）；University of Southern California（南加州大学）
We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated heterogeneous robots in a rich multi-room home environment. We provide an integrated learning framework, multimodal implementations of state-of-the-art multi-agent reinforcement learning techniques, and a consistent evaluation protocol. Our experiments investigate the impact of different modalities on multi-agent learning performance. We also introduce a simple message passing me…
Safe and efficient co-planning of multiple robots in pedestrian participation environments is promising for applications. In this work, a novel multi-robot social-aware efficient cooperative planner that on the basis of off-policy multi-agent reinforcement learning (MARL) under partial dimension-varying observation and imperfect perception conditions is proposed. We adopt temporal-spatial graph (TSG)-based social encoder to better extract the importance of social relation between each robot and the pedestrians in its field of view (FOV). Also, we introduce K-step lookahead reward setting in mu…
In Multi-Agent Reinforcement Learning, communication is critical to encourage cooperation among agents. Communication in realistic wireless networks can be highly unreliable due to network conditions varying with agents' mobility, and stochasticity in the transmission process. We propose a framework to learn practical communication strategies by addressing three fundamental questions: (1) When: Agents learn the timing of communication based on not only message importance but also wireless channel conditions. (2) What: Agents augment message contents with wireless network measurements to better…
Media and Data Science Research Lab, Adobe（Adobe媒体与数据科学研究实验室）；IIT Kharagpur, India（印度理工学院卡格拉普尔分校）
In social dilemma situations, individual rationality leads to sub-optimal group outcomes. Several human engagements can be modeled as a sequential (multi-step) social dilemmas. However, in contrast to humans, Deep Reinforcement Learning agents trained to optimize individual rewards in sequential social dilemmas converge to selfish, mutually harmful behavior. We introduce a status-quo loss (SQLoss) that encourages an agent to stick to the status quo, rather than repeatedly changing its policy. We show how agents trained with SQLoss evolve cooperative behavior in several social dilemma matrix ga…
University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
This article reviews recent advances in multi-agent reinforcement learning algorithms for large-scale control systems and communication networks, which learn to communicate and cooperate. We provide an overview of this emerging field, with an emphasis on the decentralized setting under different coordination protocols. We highlight the evolution of reinforcement learning algorithms from single-agent to multi-agent systems, from a distributed optimization perspective, and conclude with future directions and challenges, in the hope to catalyze the growing synergy among distributed optimization, …
MASH: Cooperative-Heterogeneous Multi-Agent Reinforcement Learning for Single Humanoid Robot Locomotion
Faculty of Robot Science and Engineering, Northeastern University（机器人科学与工程学院，东北大学）；Guangdong Key Laboratory of Intelligent Morphing Mechanisms and Adaptive Robotics and School of Intelligence Science and Engineering, the Harbin Institute of Technology Shenzhen（智能变形机制与适应机器人重点实验室和智能科学与工程学院，哈尔滨工业大学深圳）
Recently, multiagent deep reinforcement learning (DRL) has received increasingly wide attention. Existing multiagent DRL algorithms are inefficient when facing with the non-stationarity due to agents update their policies simultaneously in stochastic cooperative environments. This paper extends the recently proposed weighted double estimator to the multiagent domain and propose a multiagent DRL framework, named weighted double deep Q-network (WDDQN). By utilizing the weighted double estimator and the deep neural network, WDDQN can not only reduce the bias effectively but also be extended to sc…
Gatsby Computational Neuroscience Unit, University College London（伦敦大学学院盖茨比计算神经科学单位）；Max Planck Institute for Biological Cybernetics（马克斯·普朗克生物控制论研究所）
We investigate how reinforcement learning agents can learn to cooperate. Drawing inspiration from human societies, in which successful coordination of many individuals is often facilitated by hierarchical organisation, we introduce Feudal Multi-agent Hierarchies (FMH). In this framework, a 'manager' agent, which is tasked with maximising the environmentally-determined reward function, learns to communicate subgoals to multiple, simultaneously-operating, 'worker' agents. Workers, which are rewarded for achieving managerial subgoals, take concurrent actions in the world. We outline the structure…
Energy-harvesting and wireless power transfer in cooperative relaying networks have recently attracted a considerable amount of research attention. Most of the existing work on this topic however focuses on Rayleigh fading channels which represents outdoor environments. Unlike these studies, in this paper we analyze the performance of wireless power transfer in two-hop decode-and-forward (DF) cooperative relaying systems in indoor channels characterized by log-normal fading. Three well-known energy-harvesting protocols are considered in our evaluations: a) time-switching relaying (TSR), b) pow…
We consider a cognitive wireless powered communication network (CWPCN) sharing the spectrum with a primary network who faces security threats from eavesdroppers (EAVs). We propose a new cooperative protocol for the wireless powered secondary users (SU) to cooperate with the primary user (PU). In the protocol, the SUs first harvest energy from the power signals transmitted by the cognitive hybrid access point during the wireless power transfer (WPT) phase, and then use the harvested energy to interfere with the EAVs and gain transmission opportunities at the same time during the wireless inform…
In this paper, a novel cooperation-aided localization and tracking approach, suitable for terahertz (THz) wireless systems is presented. It combines an angle of arrival (AoA) tracking algorithm with the two-way time of arrival method, in order to accurately track the user equipment (UE) position and reduce the deafness caused by the estimation errors of the tracking algorithms. This algorithm can be used by one base station (BS) to estimate the UE position, or by multiple BSs, that cooperate with each other to increase the accuracy of the estimations, as well as the probability of successful e…
Formation Control for CRLB-Optimal Cooperative Sensing in Low-Altitude Wireless Networks
Southern University of Science and Technology（南方科技大学）；The Education University of Hong Kong（香港教育大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Liaoning Petrochemical University（辽宁石油化工大学）；Shenzhen University（深圳大学）
Carnegie Mellon University（卡内基梅隆大学）；InfoTech Labs, Toyota Motor North America R&D（丰田北美研发公司InfoTech实验室）
Sensor-based perception on vehicles are becoming prevalent and important to enhance the road safety. Autonomous driving systems use cameras, LiDAR, and radar to detect surrounding objects, while human-driven vehicles use them to assist the driver. However, the environmental perception by individual vehicles has the limitations on coverage and/or detection accuracy. For example, a vehicle cannot detect objects occluded by other moving/static obstacles. In this paper, we present a cooperative perception scheme with deep reinforcement learning to enhance the detection accuracy for the surrounding…
In this paper, we investigate centralized scheduling strategies for cooperative incremental redundancy retransmissions in the slow-fading half-duplex multiple access multiple relay channel. Time Division Multiple Access is assumed for the sources and the relays. Sources transmit successively in time slots for the first phase. The second phase consists of a limited number of time slots for retransmissions. In each time slot, the destination schedules a node (being a relay or a source) to retransmit, conditional on the knowledge of the correctly decoded source sets of each node (which is itself …
This paper presents a new cooperative wireless communication network strategy that incorporates energy cooperation and data cooperation. The model establishment, design goal formulations, and algorithms for throughput maximization of the proposed protocol are presented and illustrated using a three-node network with two energy harvesting (EH) user nodes and a destination node. Transmission models are established from the performance analysis for a total of four scenarios. Based on the models, we seek to find optimal energy management strategies by jointly optimizing time allocation for each us…
Maximum distance separable erasure coding has been introduced in wireless networks based on random medium access protocols in order to recover collided and erased packets. So, this help to avoid retransmission process which weaken the network throughput and prolong the overall propagation delay. However, erasure coding may degrade the network performances by causing overload traffic due to added redundancy (parity) and also when extending the propagation delay due to additional calculations and encoding/decoding process time which can lead quickly the network into saturation. Here, the redunda…
In this paper, we are concerned with the problem of estimating the speed of an RF jammer that moves towards a group/platoon of moving wireless communicating nodes. In our system model, the group of nodes receives an information signal from a master node, that they want to decode, while the Radio Frequency (RF) jammer desires to disrupt this communication as it approaches them. For this system model, we propose first a transmission scheme where the master node remains silent for a time period while it transmits in a subsequent slot. Second, we develop a joint data and jamming estimation algorit…
In this work, we investigate the impact of two cooperative unmanned aerial vehicle (UAV)-based jammers on the secrecy performance of a ground wireless network in the presence of an eavesdropper. For that purpose, we investigate the secrecy-area related metrics, Jamming Coverage and Jamming Efficiency. Moreover, we propose a hybrid metric, the so-called Weighted Secrecy Coverage (WSC) and a virtual distributed multiple-input-multiple-output (MIMO)-based zero-forcing precoding scheme to avoid the jamming effects on the legitimate receiver. For evaluating these metrics, we derive a closed-form po…
In this paper, wireless power transfer and cooperative jamming (CJ) are combined to enhance physical security in public transportation networks. First, a new secure system model with both fixed and mobile jammers is proposed to guarantee secrecy in the worst-case scenario. All jammers are endowed with energy harvesting (EH) capability. Following this, two CJ based schemes, namely B-CJ-SRM and B-CJ-TPM, are proposed, where SRM and TPM are short for secrecy rate maximization and transmit power minimization, respectively. They respectively maximize the secrecy rate (SR) with transmit power constr…
This paper investigates a multiple unmanned aerial vehicles (UAVs) enabled cooperative secure transmission scheme in the presence of multiple potential eavesdroppers. Specifically, multiple source UAVs (SUAVs) send confidential information to multiple legitimate ground users and in the meantime multiple jamming UAVs (JUAVs) cooperatively transmit interference signals to multiple eavesdroppers in order to improve the legitimate users' achievable secrecy rate. By taking into account the limited energy budget of UAV, our goal is to maximize the system secrecy energy efficiency (SEE), namely the a…
In this manuscript, we propose an alternating optimization framework to maximize the energy efficiency of a backscatter-enabled cooperative Non-orthogonal multiple access (NOMA) system by optimizing the transmit power of the source, power allocation coefficients (PAC), and power of the relay node under imperfect successive interference cancellation (SIC) decoding. A three-stage low-complexity energy-efficient alternating optimization algorithm is introduced which optimizes the transmit power, PAC, and relay power by considering the quality of service (QoS), power budget, and cooperation constr…
Network localization is capable of providing accurate and ubiquitous position information for numerous wireless applications. This paper studies the accuracy of cooperative network localization in large-scale wireless networks. Based on a decomposition of the equivalent Fisher information matrix (EFIM), we develop a random-walk-inspired approach for the analysis of EFIM, and propose a position information routing interpretation of cooperative network localization. Using this approach, we show that in large lattice and stochastic geometric networks, when anchors are uniformly distributed, the a…
