We consider an inverse reinforcement learning problem involving us versus an enemy radar equipped with a Bayesian tracker. By observing the emissions of the enemy radar,how can we identify if the radar is cognitive (constrained utility maximizer)? Given the observed sequence of actions taken by the enemy's radar, we consider three problems: (i) Are the enemy radar's actions (waveform choice, beam scheduling) consistent with constrained utility maximization? If so how can we estimate the cognitive radar's utility function that is consistent with its actions. We formulate and solve the problem in terms of the spectra (eigenvalues) of the state and observation noise covariance matrices, and the algebraic Riccati equation. (ii) How to construct a statistical test for detecting a cognitive radar (constrained utility maximization) when we observe the radar's actions in noise or the radar observes our probe signal in noise? We propose a statistical detector with a tight Type-II error bound. (iii) How can we optimally probe (interrogate) the enemy's radar by choosing our state to minimize the Type-II error of detecting if the radar is deploying an economic rational strategy, subject to a constraint on the Type-I detection error? We present a stochastic optimization algorithm to optimize our probe signal. The main analysis framework used in this paper is that of revealed preferences from microeconomics.
Cornell University（康奈尔大学）；Lockheed Martin Advanced Technology Laboratories（洛克希德·马丁先进技术实验室）
This paper considers meta-cognitive radars in an adversarial setting. A cognitive radar optimally adapts its waveform (response) in response to maneuvers (probes) of a possibly adversarial moving target. A meta-cognitive radar is aware of the adversarial nature of the target and seeks to mitigate the adversarial target. How should the meta-cognitive radar choose its responses to sufficiently confuse the adversary trying to estimate the radar's utility function? This paper abstracts the radar's meta-cognition problem in terms of the spectra (eigenvalues) of the state and observation noise covar…
Cornell University（康奈尔大学）；DEVCOM Army Research Laboratory（美国陆军作战能力发展司令部陆军研究实验室）；U.S Army Research Laboratory（美国陆军研究实验室）
Consider a target being tracked by a cognitive radar network. If the target can intercept some radar network emissions, how can it detect coordination among the radars? By 'coordination' we mean that the radar emissions satisfy Pareto optimality with respect to multi-objective optimization over each radar's utility. This paper provides a novel multi-objective inverse reinforcement learning approach which allows for both detection of such Pareto optimal ('coordinating') behavior and subsequent reconstruction of each radar's utility function, given a finite dataset of radar network emissions. Th…
Cornell University（康奈尔大学）；Lockheed Martin Advanced Technology Laboratories（洛克希德·马丁先进技术实验室）
A cognitive radar is a constrained utility maximizer that adapts its sensing mode in response to a changing environment. If an adversary can estimate the utility function of a cognitive radar, it can determine the radar's sensing strategy and mitigate the radar performance via electronic countermeasures (ECM). This paper discusses how a cognitive radar can {\em hide} its strategy from an adversary that detects cognition. The radar does so by transmitting purposefully designed sub-optimal responses to spoof the adversary's Neyman-Pearson detector. We provide theoretical guarantees by ensuring t…
When should an online reinforcement learning-based frequency agile cognitive radar be expected to outperform a rule-based adaptive waveform selection strategy? We seek insight regarding this question by examining a dynamic spectrum access scenario, in which the radar wishes to transmit in the widest unoccupied bandwidth during each pulse repetition interval. Online learning is compared to a fixed rule-based sense-and-avoid strategy. We show that given a simple Markov channel model, the problem can be examined analytically for simple cases via stochastic dominance. Additionally, we show that fo…
An intelligent radar resource management is an essential milestone in the development of a cognitive radar system. The quality of service based resource allocation model (Q-RAM) is a framework allowing for intelligent decision making but classical solutions seem insufficient for real-time application in a modern radar system. In this paper, we present a solution for the Q-RAM radar resource management problem using deep reinforcement learning considerably improving on runtime performance.
A Fast Anti-Jamming Cognitive Radar Deployment Algorithm Based on Reinforcement Learning
School of Mathematical Sciences University of Chinese Academy of Sciences（数学科学学院 中国科学院大学）；Information Science Academy China Electronics Technology Group Corporation（信息科学院 中国电子科技集团）
Virginia Tech（弗吉尼亚理工大学）；U.S Army Research Laboratory（美国陆军研究实验室）
This paper describes a sequential, or online, learning scheme for adaptive radar transmissions that facilitate spectrum sharing with a non-cooperative cellular network. First, the interference channel between the radar and a spatially distant cellular network is modeled. Then, a linear Contextual Bandit (CB) learning framework is applied to drive the radar's behavior. The fundamental trade-off between exploration and exploitation is balanced by a proposed Thompson Sampling (TS) algorithm, a pseudo-Bayesian approach which selects waveform parameters based on the posterior probability that a spe…
Institute of Digital Communication Systems, Ruhr University Bochum（鲁尔大学波鸿分校数字通信系统研究所）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；DR2I-IPSA（DR2I-IPSA机构）；Università di Pisa（比萨大学）
This paper considers the problem of multi-target detection for massive multiple input multiple output (MMIMO) cognitive radar (CR). The concept of CR is based on the perception-action cycle that senses and intelligently adapts to the dynamic environment in order to optimally satisfy a specific mission. However, this usually requires a priori knowledge of the environmental model, which is not available in most cases. We propose a reinforcement learning (RL) based algorithm for cognitive multi-target detection in the presence of unknown disturbance statistics. The radar acts as an agent that con…
Cornell University（康奈尔大学）；Lockheed Martin Advanced Technology Laboratories（洛克希德·马丁高级技术实验室）
This paper considers adaptive radar electronic counter-counter measures (ECCM) to mitigate ECM by an adversarial jammer. Our ECCM approach models the jammer-radar interaction as a Principal Agent Problem (PAP), a popular economics framework for interaction between two entities with an information imbalance. In our setup, the radar does not know the jammer's utility. Instead, the radar learns the jammer's utility adaptively over time using inverse reinforcement learning. The radar's adaptive ECCM objective is two-fold (1) maximize its utility by solving the PAP, and (2) estimate the jammer's ut…
In this paper, dynamic non-cooperative coexistence between a cognitive pulsed radar and a nearby communications system is addressed by applying nonlinear value function approximation via deep reinforcement learning (Deep RL) to develop a policy for optimal radar performance. The radar learns to vary the bandwidth and center frequency of its linear frequency modulated (LFM) waveforms to mitigate mutual interference with other systems and improve target detection performance while also maintaining sufficient utilization of the available frequency bands required for a fine range resolution. We de…
This paper describes some key ideas and applications of cognitive radars, highlighting the limits and the path forward. Cognitive radars are systems based on the perception-action cycle of cognition that sense the environment, learn from it relevant information about the target and the background, then adapt the radar sensor to optimally satisfy the needs of their mission according to a desired goal. The concept of cognitive radar was introduced originally for active radar only. In this paper we describe how this paradigm can be applied also to passive radar. In particular, we describe (i) cog…
Cognitive sensing refers to a reconfigurable sensor that dynamically adapts its sensing mechanism by using stochastic control to optimize its sensing resources. For example, cognitive radars are sophisticated dynamical systems; they use stochastic control to sense the environment, learn from it relevant information about the target and background, then adapt the radar sensor to satisfy the needs of their mission. The last two decades have witnessed intense research in cognitive/adaptive radars.This paper discusses addresses the next logical step, namely inverse cognitive sensing. By observing …
This paper investigates the anti-jamming performance of a cognitive radar under a partially observable Markov decision process (POMDP) model. First, we obtain an explicit expression for uncertainty of jammer dynamics, which paves the way for illuminating the performance metric of probability of being jammed for the radar beyond a conventional signal-to-noise ratio ($\mathsf{SNR}$) based analysis. Considering two frequency hopping strategies developed in the framework of reinforcement learning (RL), this performance metric is analyzed with deep Q-network (DQN) and long short term memory (LSTM) …
We study how a cognitive radar can mask (hide) its cognitive ability from an adversarial jamming device. Specifically, if the radar optimally adapts its waveform based on adversarial target maneuvers (probes), how should the radar choose its waveform parameters (response) so that its utility function cannot be recovered by the adversary. This paper abstracts the radar's cognition masking problem in terms of the spectra (eigenvalues) of the state and observation noise covariance matrices, and embeds the algebraic Riccati equation into an economics-based utility maximization setup. Given an obse…
Cornell University（康奈尔大学）；Information Systems Laboratories, Inc.（信息系统实验室公司）；University of Dayton Research Institute（代顿大学研究院）；Air Force Research Laboratory（空军研究实验室）
This paper considers three inter-related adversarial inference problems involving cognitive radars. We first discuss inverse tracking of the radar to estimate the adversary's estimate of us based on the radar's actions and calibrate the radar's sensing accuracy. Second, using revealed preference from microeconomics, we formulate a non-parametric test to identify if the cognitive radar is a constrained utility maximizer with signal processing constraints. We consider two radar functionalities, namely, beam allocation and waveform design, with respect to which the cognitive radar is assumed to m…
Completely decentralized Multi-Player Bandit models have demonstrated high localization accuracy at the cost of long convergence times in cognitive radar networks. Rather than model each radar node as an independent learner, entirely unable to swap information with other nodes in a network, in this work we construct a "central coordinator" to facilitate the exchange of information between radar nodes. We show that in interference-limited spectrum, where the signal to interference plus noise (SINR) ratio for the available bands may vary by location, a cognitive radar network (CRN) is able to us…
Random stepped frequency (RSF) radar, which transmits random-frequency pulses, can suppress the range ambiguity, improve convert detection, and possess excellent electronic counter-countermeasures (ECCM) ability [1]. In this paper, we apply a sparse recovery method to estimate the range and Doppler of targets. We also propose a cognitive mechanism for RSF radar to further enhance the performance of the sparse recovery method. The carrier frequencies of transmitted pulses are adaptively designed in response to the observed circumstance. We investigate the criterion to design carrier frequencies…
Cognitive multiple-input multiple-output (MIMO) radar is capable of adjusting system parameters adaptively by sensing and learning in complex dynamic environment. Beamforming performance of MIMO radar is guided by both beamforming weight coefficients and the transceiver configuration. We propose a cognitive-driven MIMO array design where both the beamforming weights and the transceiver configuration are adaptively and concurrently optimized under different environmental conditions. The perception-action cycle involves data collection of full virtual array, covariance reconstruction and joint d…
Learning Goal-based Movement via Motivational-based Models in Cognitive Mobile Robots
University of Campinas（坎皮纳斯大学）；São Paulo State University（圣保罗州立大学）；Artificial Intelligence and Cognitive Architectures Hub (H.IAAC), University of Campinas（坎皮纳斯大学人工智能与认知架构中心（H.IAAC））
We model a radar network as an adversarial bandit problem, where the environment pre-selects reward sequences for each of several actions available to the network. This excludes environments which vary rewards in response to the learner's actions. Adversarial environments include those with third party emitters which enter and exit the environment according to some criteria which does not depend on the radar network. The network consists of several independent radar nodes, which attempt to attain the highest possible SINR in each of many time steps. We show that in such an environment, simple …
The concept of cognitive radar (CR) enables radar systems to achieve intelligent adaption to a changeable environment with feedback facility from receiver to transmitter. However, the implementation of CR in a fast-changing environment usually requires a well-known environmental model. In our work, we stress the learning ability of CR in an unknown environment using a combination of CR and reinforcement learning (RL), called RL-CR. Less or no model of the environment is required. We also apply the general RL-CR to a specific problem of automotive radar spectrum allocation to mitigate mutual in…
Modern radars often adopt multi-carrier waveform which has been widely discussed in the literature. However, with the development of civil communication, more and more spectrum resource has been occupied by communication networks. Thus, avoiding the interference from communication users is an important and challenging task for the application of multi-carrier radar. In this paper, a novel frequency allocation strategy based on the historical experiences is proposed, which is formulated as a Markov decision process (MDP). In a decision step, the multi-carrier radar needs to choose more than one…
Direction of arrival (DoA) estimation of targets improves with the number of elements employed by a phased array radar antenna. Since larger arrays have high associated cost, area and computational load, there is recent interest in thinning the antenna arrays without loss of far-field DoA accuracy. In this context, a cognitive radar may deploy a full array and then select an optimal subarray to transmit and receive the signals in response to changes in the target environment. Prior works have used optimization and greedy search methods to pick the best subarrays cognitively. In this paper, we …
We present a cognitive prototype that demonstrates a colocated, frequency-division-multiplexed, multiple-input multiple-output (MIMO) radar which implements both temporal and spatial sub-Nyquist sampling. The signal is sampled and recovered via the Xampling framework. Cognition is due to the fact that the transmitter adapts its signal spectrum by emitting only those subbands that the receiver samples and processes. Real-time experiments demonstrate sub-Nyquist MIMO recovery of target scenes with 87.5% spatio-temporal bandwidth reduction and signal-to-noise-ratio of -10 dB.
School of Computing and Information Science（计算与信息科学学院）；Florida International University（佛罗里达国际大学）
Interactive Machine Learning is concerned with creating systems that operate in environments alongside humans to achieve a task. A typical use is to extend or amplify the capabilities of a human in cognitive or physical ways, requiring the machine to adapt to the users' intentions and preferences. Often, this takes the form of a human operator providing some type of feedback to the user, which can be explicit feedback, implicit feedback, or a combination of both. Explicit feedback, such as through a mouse click, carries a high cognitive load. The focus of this study is to extend the current st…
This work addresses the coexistence problem for radar networks. Specifically, we model a network of cooperative, independent, and non-communicating radar nodes which must share resources within the network as well as with non-cooperative nearby emitters. We approach this problem using online Machine Learning (ML) techniques. Online learning approaches are specifically preferred due to the fact that each radar node has no prior knowledge of the environment nor of the positions of the other radar nodes, and due to the sequential nature of the problem. For this task we specifically select the mul…
New generation of radar systems will need to coexist with other radio frequency (RF) systems, anticipating their behavior and reacting appropriately to avoid interference. In light of this requirement, this paper designs, implements, and evaluates the performance of phase-only sequences (with constant power) for intelligent spectrum utilization using the custom built cognitive Multiple Input Multiple Output (MIMO) radar prototype. The proposed transmit waveforms avoid the frequency bands occupied by narrowband interferers or communication links, while simultaneously have a small cross-correlat…
Innate-Values-driven Reinforcement Learning based Cooperative Multi-Agent Cognitive Modeling
Intelligent Social Systems and Swarm Robotics Lab (IS$^3$R)（智能社会系统与群体机器人实验室）；Computer Science and Information Systems Department（计算机科学与信息系统系）；Bradley University（布拉德利大学）
Virginia Tech（弗吉尼亚理工大学）；US Army Research Laboratory（美国陆军研究实验室）
A key component of cognitive radar is the ability to generalize, or achieve consistent performance across a range of sensing environments, since aspects of the physical scene may vary over time. This presents a challenge for learning-based waveform selection approaches, since transmission policies which are effective in one scene may be highly suboptimal in another. We address this problem by strategically biasing a learning algorithm by exploiting high-level structure across tracking instances, referred to as meta-learning. In this work, we develop an online meta-learning approach for wavefor…
Cognitive radars are systems that rely on learning through interactions of the radar with the surrounding environment. To realize this, radar transmit parameters can be adapted such that they facilitate some downstream task. This paper proposes the use of deep reinforcement learning (RL) to learn policies for gain control under the object detection task. The YOLOv3 single-shot object detector is used for the downstream task and will be concurrently used alongside the RL agent. Furthermore, a synthetic dataset is introduced which models the radar environment with use of the Grand Theft Auto V g…
An important problem in cognitive radar is to enhance the estimation performance of the system by a joint design of its probing signal and receive filter using the a priori information on interference. In such cases, the knowledge of interference statistics (particularly the covariance) plays a vital role in an effective design of the radar waveforms. In most practical scenarios, however, the received signal and interference statistics are available subject to some uncertainty. An extreme manifestation of this practical observation occurs for radars employing one-bit receivers, where only a no…
Nowadays, mutual interference among automotive radars has become a problem of wide concern. In this paper, a decentralized spectrum allocation approach is presented to avoid mutual interference among automotive radars. Although decentralized spectrum allocation has been extensively studied in cognitive radio sensor networks, two challenges are observed for automotive sensors using radar. First, the allocation approach should be dynamic as all radars are mounted on moving vehicles. Second, each radar does not communicate with the others so it has quite limited information. A machine learning te…
This paper aims to design a set of transmitting waveforms in cognitive colocated Multi-Input Multi-Output (MIMO) radar systems considering the simultaneous minimization of spatial- and the range- Integrated Sidelobe Level Ratio (ISLR). The design problem is formulated as a bi-objective Pareto optimization under practical constraints on the waveforms, namely total transmit power, peak-to-average-power ratio (PAR), constant modulus, and discrete phase alphabet. A Coordinate Descent (CD) based approach is proposed, in which at every single variable update of the algorithm we obtain the solution o…
A cognitive beamforming algorithm for colocated MIMO radars, based on Reinforcement Learning (RL) framework, is proposed. We analyse an RL-based optimization protocol that allows the MIMO radar, i.e. the \textit{agent}, to iteratively sense the unknown environment, i.e. the radar scene involving an unknown number of targets at unknown angular positions, and consequently, to synthesize a set of transmitted waveforms whose related beam patter is tailored on the acquired knowledge. The performance of the proposed RL-based beamforming algorithm is assessed through numerical simulations in terms of…
Growing number of wireless devices and networks has increased the demand for the scarce resource, radio spectrum. Next generation communication technologies, such as Cognitive Radio provides a promising solution to efficiently utilize radio spectrum whilst delivering improved data communication rate, service, and security. A cognitive radio system will be able to sense the availability of radio frequencies, analyze the condition of the sensed channels, and decide the best option for optimal communication. To select the best option out of the overwhelming amount of information, a channel rankin…
