The advent of artificial intelligence technology paved the way of many researches to be made within air combat sector. Academicians and many other researchers did a research on a prominent research direction called autonomous maneuver decision of UAV. Elaborative researches produced some outcomes, but decisions that include Reinforcement Learning(RL) came out to be more efficient. There have been many researches and experiments done to make an agent reach its target in an optimal way, most prominent are Genetic Algorithm(GA) , A star, RRT and other various optimization techniques have been used. But Reinforcement Learning is the well known one for its success. In DARPHA Alpha Dogfight Trials, reinforcement learning prevailed against a real veteran F16 human pilot who was trained by Boeing. This successor model was developed by Heron Systems. After this accomplishment, reinforcement learning bring tremendous attention on itself. In this research we aimed our UAV which has a dubin vehicle dynamic property to move to the target in two dimensional space in an optimal path using Twin Delayed Deep Deterministic Policy Gradients (TD3) and used in experience replay Hindsight Experience Replay(HER).We did tests on two different environments and used simulations.
Aeronautics Engineering College, Air Force Engineering University（空军工程大学航空工程学院）
Reinforcement learning is an effective way to solve the decision-making problems. It is a meaningful and valuable direction to investigate autonomous air combat maneuver decision-making method based on reinforcement learning. However, when using reinforcement learning to solve the decision-making problems with sparse rewards, such as air combat maneuver decision-making, it costs too much time for training and the performance of the trained agent may not be satisfactory. In order to solve these problems, the method based on curriculum learning is proposed. First, three curricula of air combat m…
A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Aircraft Design and Research Institute (SADRI)（沈阳飞机设计研究所）
The Intelligent decision of the unmanned combat aerial vehicle (UCAV) has long been a challenging problem. The conventional search method can hardly satisfy the real-time demand during high dynamics air combat scenarios. The reinforcement learning (RL) method can significantly shorten the decision time via using neural networks. However, the sparse reward problem limits its convergence speed and the artificial prior experience reward can easily deviate its optimal convergent direction of the original task, which raises great difficulties for the RL air combat application. In this paper, we pro…
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
Unmanned aircraft systems can perform some more dangerous and difficult missions than manned aircraft systems. In some highly complicated and changeable tasks, such as air combat, the maneuvering decision mechanism is required to sense the combat situation accurately and make the optimal strategy in real-time. This paper presents a formulation of a 3-D one-on-one air combat maneuvering problem and an approximate dynamic programming approach for computing an optimal policy on autonomous maneuvering decision making. The aircraft learns combat strategies in a Reinforcement Leaning method, while s…
Explaining Strategic Decisions in Multi-Agent Reinforcement Learning for Aerial Combat Tactics
Dalle Molle Institute for Artificial Intelligence (IDSIA) - SUPSI/USI（达勒莫尔人工智能研究所（IDSIA）- 瑞士南部应用科学与艺术大学/瑞士意大利语区大学）；armasuisse Science and Technology（瑞士联邦国防采购局科学与技术）
Institute for Advanced Studies（高级研究院）；Aeronautics Institute of Technology（航空技术学院）
This work aims to provide an engagement decision support tool for Beyond Visual Range (BVR) air combat in the context of Defensive Counter Air (DCA) missions. In BVR air combat, engagement decision refers to the choice of the moment the pilot engages a target by assuming an offensive stance and executing corresponding maneuvers. To model this decision, we use the Brazilian Air Force's Aerospace Simulation Environment (Ambiente de Simulação Aeroespacial - ASA in Portuguese), which generated 3,729 constructive simulations lasting 12 minutes each and a total of 10,316 engagements. We analyzed all…
Evolutionary Enhanced Multi-Agent Reinforcement Learning for Cooperative Air Combat
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Normally a decision support system is build to solve problem where multi-criteria decisions are involved. The knowledge base is the vital part of the decision support containing the information or data that is used in decision-making process. This is the field where engineers and scientists have applied several intelligent techniques and heuristics to obtain optimal decisions from imprecise information. In this paper, we present a hybrid neuro-genetic learning approach for the adaptation a Mamdani fuzzy inference system for the Tactical Air Combat Decision Support System (TACDSS). Some simulat…
Artificial intelligence (AI) has enormous potential to improve Air Force pilot training by providing actionable feedback to pilot trainees on the quality of their maneuvers and enabling instructor-less flying familiarization for early-stage trainees in low-cost simulators. Historically, AI challenges consisting of data, problem descriptions, and example code have been critical to fueling AI breakthroughs. The Department of the Air Force-Massachusetts Institute of Technology AI Accelerator (DAF-MIT AI Accelerator) developed such an AI challenge using real-world Air Force flight simulator data. …
Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering
Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA)（达勒·莫尔人工智能研究所（IDSIA））；USI-SUPSI（瑞士意大利语区大学-瑞士意大利语区应用科学与艺术大学）；Armasuisse Science + Technology（瑞士联邦国防采购局科技中心）
DRG-MAPPO: Hierarchical Dynamic Role-Graph Multi-Agent Reinforcement Learning for Cooperative Air Combat
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, CASIA（中国科学院自动化研究所复杂系统认知与决策智能重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Istanbul Technical University（伊斯坦布尔技术大学）；Koç University（科奇大学）；HARFANG 3D NWNC
The advent of deep learning (DL) gave rise to significant breakthroughs in Reinforcement Learning (RL) research. Deep Reinforcement Learning (DRL) algorithms have reached super-human level skills when applied to vision-based control problems as such in Atari 2600 games where environment states were extracted from pixel information. Unfortunately, these environments are far from being applicable to highly dynamic and complex real-world tasks as in autonomous control of a fighter aircraft since these environments only involve 2D representation of a visual world. Here, we present a semi-realistic…
Institute for Advanced Studies（高级研究所）；Aeronautics Institute of Technology（航空技术学院）
This work compares supervised machine learning methods using reliable data from constructive simulations to estimate the most effective moment for launching missiles during air combat. We employed resampling techniques to improve the predictive model, analyzing accuracy, precision, recall, and f1-score. Indeed, we could identify the remarkable performance of the models based on decision trees and the significant sensitivity of other algorithms to resampling techniques. The models with the best f1-score brought values of 0.379 and 0.465 without and with the resampling technique, respectively, w…
Institute for Systems Studies and Analyses（系统研究与分析研究所）
The paper is a half-way between the agent technology and the mathematical reasoning to model tactical decision making tasks. These models are applied to air defense (AD) domain for command and control (C2). It also addresses the issues related to evaluation of agents. The agents are designed and implemented using the agent-programming paradigm. The agents are deployed in an air combat simulated environment for performing the tasks of C2 like electronic counter counter measures, threat assessment, and weapon allocation. The simulated AD system runs without any human intervention, and represents…
Enhancing Aerial Combat Tactics through Hierarchical Multi-Agent Reinforcement Learning
Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA), USI-SUPSI（瑞士智能系统研究所（IDSIA）及USI-SUPSI）；Armasuisse Science + Technology（Armasuisse科学与技术）
