Recent years have seen an explosion of interest in autonomous cyber defence agents trained to defend computer networks using deep reinforcement learning. These agents are typically trained in cyber gym environments using dense, highly engineered reward functions which combine many penalties and incentives for a range of (un)desirable states and costly actions. Dense rewards help alleviate the challenge of exploring complex environments but risk biasing agents towards suboptimal and potentially riskier solutions, a critical issue in complex cyber environments. We thoroughly evaluate the impact of reward function structure on learning and policy behavioural characteristics using a variety of sparse and dense reward functions, two well-established cyber gyms, a range of network sizes, and both policy gradient and value-based RL algorithms. Our evaluation is enabled by a novel ground truth evaluation approach which allows directly comparing between different reward functions, illuminating the nuanced inter-relationships between rewards, action space and the risks of suboptimal policies in cyber environments. Our results show that sparse rewards, provided they are goal aligned and can be encountered frequently, uniquely offer both enhanced training reliability and more effective cyber defence agents with lower-risk policies. Surprisingly, sparse rewards can also yield policies that are better aligned with cyber defender goals and make sparing use of costly defensive actions without explicit reward-based numerical penalties.
Within recent times, cybercriminals have curated a variety of organised and resolute cyber attacks within a range of cyber systems, leading to consequential ramifications to private and governmental institutions. Current security-based automation and orchestrations focus on automating fixed purpose and hard-coded solutions, which are easily surpassed by modern-day cyber attacks. Research within Automated Cyber Defence will allow the development and enabling intelligence response by autonomously defending networked systems through sequential decision-making agents. This article comprehensively …
The Alan Turing Institute（艾伦·图灵研究所）；Imperial College London（帝国理工学院）
Cybercriminals are rapidly developing new malicious tools that leverage artificial intelligence (AI) to enable new classes of adaptive and stealthy attacks. New defensive methods need to be developed to counter these threats. Some cybersecurity professionals are speculating AI will enable corresponding new classes of active cyber defence measures -- is this realistic, or currently mostly hype? The Alan Turing Institute, with expert guidance from the UK National Cyber Security Centre and Defence Science Technology Laboratory, published a research roadmap for AI for ACD last year. This position …
Learning to Communicate in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence
Electrical and Computer Engineering Department, Royal Military College of Canada（加拿大皇家军事学院电子与计算机工程系）；Defence Research and Development Canada（加拿大国防研究发展署）
本文提出在Cyber Operations Research Gym中利用可微分智能体间学习算法训练防御智能体，使其在部分可观测网络环境中学习低成本通信与防御战术策略，从而提升自主网络防御的协同决策能力。
Building Better Environments for Autonomous Cyber Defence
The Alan Turing Institute（艾伦·图灵研究所）；University College London（伦敦大学学院）；Cornell University（康奈尔大学）；Vanderbilt University（范德比尔特大学）；Microsoft（微软）；NSA（美国国家安全局）
Learning Communication Between Heterogeneous Agents in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence
Department of Electrical and Computer Engineering, Royal Military College of Canada（加拿大皇家军事学院电子与计算机工程系）；Defence Research and Development Canada（加拿大国防研究与发展加拿大）；Department of Computer Engineering and Software Engineering, Polytechnique Montréal（蒙特利尔理工学院计算机工程与软件工程系）
The University of Melbourne（墨尔本大学）；Defence Science and Technology Group（国防科学技术集团）
Despite the successful application of machine learning (ML) in a wide range of domains, adaptability---the very property that makes machine learning desirable---can be exploited by adversaries to contaminate training and evade classification. In this paper, we investigate the feasibility of applying a specific class of machine learning algorithms, namely, reinforcement learning (RL) algorithms, for autonomous cyber defence in software-defined networking (SDN). In particular, we focus on how an RL agent reacts towards different forms of causative attacks that poison its training process, includ…
Defence Research and Development Canada（加拿大国防研究与发展部）；Queens University（女王大学）
Given the success of reinforcement learning (RL) in various domains, it is promising to explore the application of its methods to the development of intelligent and autonomous cyber agents. Enabling this development requires a representative RL training environment. To that end, this work presents CyGIL: an experimental testbed of an emulated RL training environment for network cyber operations. CyGIL uses a stateless environment architecture and incorporates the MITRE ATT&CK framework to establish a high fidelity training environment, while presenting a sufficiently abstracted interface to en…
The University of Melbourne（墨尔本大学）；Defence Science and Technology Group, Australian Department of Defence（澳大利亚国防部国防科技集团）
Recent studies have demonstrated that reinforcement learning (RL) agents are susceptible to adversarial manipulation, similar to vulnerabilities previously demonstrated in the supervised learning setting. While most existing work studies the problem in the context of computer vision or console games, this paper focuses on reinforcement learning in autonomous cyber defence under partial observability. We demonstrate that under the black-box setting, where the attacker has no direct access to the target RL model, causative attacks---attacks that target the training process---can poison RL agents…
Command and Control Engineering College, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；troop of PLA（中国人民解放军31436部队）；Academy of Military Science（军事科学院）
The existing network attack and defense method can be regarded as game, but most of the game only involves network domain, not multiple domain cyberspace. To address this challenge, this paper proposed a multiple domain cyberspace attack and defense game model based on reinforcement learning. We define the multiple domain cyberspace include physical domain, network domain and digital domain. By establishing two agents, representing the attacker and the defender respectively, defender will select the multiple domain actions in the multiple domain cyberspace to obtain defender's optimal reward b…
Command and Control Engineering College, Army Engineering University of PLA（中国人民解放军陆军工程大学指挥控制工程学院）；troop of PLA（中国人民解放军31436部队）；Academy of Military Science（军事科学院）
In general, multiple domain cyberspace security assessments can be implemented by reasoning user's permissions. However, while existing methods include some information from the physical and social domains, they do not provide a comprehensive representation of cyberspace. Existing reasoning methods are also based on expert-given rules, resulting in inefficiency and a low degree of intelligence. To address this challenge, we create a Knowledge Graph (KG) of multiple domain cyberspace in order to provide a standard semantic description of the multiple domain cyberspace. Following that, we propos…
Defence Science and Technology Laboratory（国防科学技术实验室）；The Alan Turing Institute（艾伦·图灵研究所）
In this paper we explore cyber security defence, through the unification of a novel cyber security simulator with models for (causal) decision-making through optimisation. Particular attention is paid to a recently published approach: dynamic causal Bayesian optimisation (DCBO). We propose that DCBO can act as a blue agent when provided with a view of a simulated network and a causal model of how a red agent spreads within that network. To investigate how DCBO can perform optimal interventions on host nodes, in order to reduce the cost of intrusions caused by the red agent. Through this we dem…
University of Technology Sydney（悉尼科技大学）；Defence Science and Technology Group（国防科技集团）
Penetration testing the organised attack of a computer system in order to test existing defences has been used extensively to evaluate network security. This is a time consuming process and requires in-depth knowledge for the establishment of a strategy that resembles a real cyber-attack. This paper presents a novel deep reinforcement learning architecture with hierarchically structured agents called HA-DRL, which employs an algebraic action decomposition strategy to address the large discrete action space of an autonomous penetration testing simulator where the number of actions is exponentia…
Data61, CSIRO Australia（澳大利亚联邦科学与工业研究组织Data61）；Cyber Security Cooperative Research Centre Australia（澳大利亚网络安全合作研究中心）；Penten Pty Ltd（Penten私人有限公司）；UNSW Sydney（悉尼新南威尔士大学）
Deception is rapidly growing as an important tool for cyber defence, complementing existing perimeter security measures to rapidly detect breaches and data theft. One of the factors limiting the use of deception has been the cost of generating realistic artefacts by hand. Recent advances in Machine Learning have, however, created opportunities for scalable, automated generation of realistic deceptions. This vision paper describes the opportunities and challenges involved in developing models to mimic many common elements of the IT stack for deception effects.
National Security Institute, Virginia Polytechnic University（弗吉尼亚理工大学国家安全研究所）；Deloitte & Touche LLP（德勤华永会计师事务所）
Reinforcement learning (RL) operating on attack graphs leveraging cyber terrain principles are used to develop reward and state associated with determination of surveillance detection routes (SDR). This work extends previous efforts on developing RL methods for path analysis within enterprise networks. This work focuses on building SDR where the routes focus on exploring the network services while trying to evade risk. RL is utilized to support the development of these routes by building a reward mechanism that would help in realization of these paths. The RL algorithm is modified to have a no…
Applying Action Masking and Curriculum Learning Techniques to Improve Data Efficiency and Overall Performance in Operational Technology Cyber Security using Reinforcement Learning
Model-Free Reinforcement Learning Control for Resilient Cyber-Physical Systems
Departmento de Ingenier\'ia El\'ectrica, Universidad de Concepci\'on, Concepci\'on, Chile (e-mail: )；Department of Electrical \& Computer Engineering, University of Alberta, Edmonton, T6G 1H9, Alberta, AB, Canada (e-mail: )；Department of Chemical；Biological Engineering, University of British Columbia, Vancouver, BC V6T 1Z3, Canada ( )；Department of Chemical \& Materials Engineering, University of Alberta, Edmonton, T6G 1H9, Alberta, AB, Canada (e-mail: )
Reinforcement Learning (RL) algorithms have led to recent successes in solving complex games, such as Atari or Starcraft, and to a huge impact in real-world applications, such as cybersecurity or autonomous driving. In the side of the drawbacks, recent works have shown how the performance of RL algorithms decreases under the influence of soft changes in the reward function. However, little work has been done about how sensitive these disturbances are depending on the aggressiveness of the attack and the learning exploration strategy. In this paper, we propose to fill this gap in the literature…
University of Texas, Austin（德克萨斯大学奥斯汀分校）；Oden Institute for Computational Engineering and Sciences（奥登计算工程与科学研究所）
Deception plays a key role in adversarial or strategic interactions for the purpose of self-defence and survival. This paper introduces a general framework and solution to address deception. Most existing approaches for deception consider obfuscating crucial information to rational adversaries with abundant memory and computation resources. In this paper, we consider deceiving adversaries with bounded rationality and in terms of expected rewards. This problem is commonly encountered in many applications especially involving human adversaries. Leveraging the cognitive bias of humans in reward e…
Autonomous Cyber Operations Gym for Training and Evaluating AI Models for Defence
CortAIx Labs, Thales Deutschland（Thales Deutschland CortAIx 实验室）；University of Osnabrück（奥斯纳布吕克大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）；University of Ulm（乌尔姆大学）
Naval Information Warfare Center Pacific（太平洋海军信息战中心）；Laboratory for Advanced Cybersecurity Research（高级网络安全研究实验室）
The performance of artificial intelligence (AI) algorithms in practice depends on the realism and correctness of the data, models, and feedback (labels or rewards) provided to the algorithm. This paper discusses methods for improving the realism and ecological validity of AI used for autonomous cyber defense by exploring the potential to use Inverse Reinforcement Learning (IRL) to gain insight into attacker actions, utilities of those actions, and ultimately decision points which cyber deception could thwart. The Tularosa study, as one example, provides experimental data of real-world techniqu…
Towards Production-Worthy Simulation for Autonomous Cyber Operations
Department of Electrical and Computer Engineering（电气与计算机工程系）；Royal Military College of Canada（加拿大皇家军事学院）；Department of Mathematics and Computer Science（数学与计算机科学系）；Defence Research and Development Canada（加拿大国防研究发展署）；Department of Computer and Software Engineering（计算机与软件工程系）
Deloitte Consulting LLC（德勤咨询有限责任公司）；National Security Institute, Virginia Tech（弗吉尼亚理工大学国家安全研究所）；Deloitte & Touche LLP（德勤会计师事务所）
Reinforcement learning (RL) has been applied to attack graphs for penetration testing, however, trained agents do not reflect reality because the attack graphs lack operational nuances typically captured within the intelligence preparation of the battlefield (IPB) that include notions of (cyber) terrain. In particular, current practice constructs attack graphs exclusively using the Common Vulnerability Scoring System (CVSS) and its components. We present methods for constructing attack graphs using notions from IPB on cyber terrain analysis of obstacles, avenues of approach, key terrain, obser…
Digitization and remote connectivity have enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain critical …
University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）
With the recent developments in artificial intelligence and machine learning, anomalies in network traffic can be detected using machine learning approaches. Before the rise of machine learning, network anomalies which could imply an attack, were detected using well-crafted rules. An attacker who has knowledge in the field of cyber-defence could make educated guesses to sometimes accurately predict which particular features of network traffic data the cyber-defence mechanism is looking at. With this information, the attacker can circumvent a rule-based cyber-defense system. However, after the …
Hierarchical Decentralized Stochastic Control for Cyber-Physical Systems
Department of Electrical Engineering and Computer Science, University of Ottawa（电气与计算机科学系，渥太华大学）；Department of Electrical Engineering, Eindhoven University of Technology (TU/e)（电气工程系，埃因霍温理工大学）；Department of Electrical Engineering, Indian Institute of Technology Madras（电气工程系，印度理工学院马德拉斯学院）
