Data Mining with Big Data in Intrusion Detection Systems: A Systematic Literature Review
The University of Western Ontario（西安大略大学）；University of Sharjah（沙迦大学）
Fadi Salo, MohammadNoor Injadat, Ali Bou Nassif, Aleksander Essex
备注：
8 Pages, 5 Figures, to be appeared in the proceedings of the International Symposium on Big Data Management and Analytics. April 25-26, 2019, Calgary, Canada
Cloud computing has become a powerful and indispensable technology for complex, high performance and scalable computation. The exponential expansion in the deployment of cloud technology has produced a massive amount of data from a variety of applications, resources and platforms. In turn, the rapid rate and volume of data creation has begun to pose significant challenges for data management and security. The design and deployment of intrusion detection systems (IDS) in the big data setting has, therefore, become a topic of importance. In this paper, we conduct a systematic literature review (SLR) of data mining techniques (DMT) used in IDS-based solutions through the period 2013-2018. We employed criterion-based, purposive sampling identifying 32 articles, which constitute the primary source of the present survey. After a careful investigation of these articles, we identified 17 separate DMTs deployed in an IDS context. This paper also presents the merits and disadvantages of the various works of current research that implemented DMTs and distributed streaming frameworks (DSF) to detect and/or prevent malicious attacks in a big data environment.
Intrusion Detection is one of major threats for organization. The approach of intrusion detection using text processing has been one of research interests which is gaining significant importance from researchers. In text mining based approach for intrusion detection, system calls serve as source for mining and predicting possibility of intrusion or attack. When an application runs, there might be several system calls which are initiated in the background. These system calls form the strong basis and the deciding factor for intrusion detection. In this paper, we mainly discuss the approach for …
In this paper, we consider the naive applications of process mining in network traffic comprehension, traffic anomaly detection, and intrusion detection. We standardise the procedure of transforming packet data into an event log. We mine multiple process models and analyse the process models mined with the inductive miner using ProM and the fuzzy miner using Disco. We compare the two types of process models extracted from event logs of differing sizes. We contrast the process models with the RFC TCP state transition diagram and the diagram by Bishop et al. We analyse the issues and challenges …
In this paper, we consider the applications of process mining in intrusion detection. We propose a novel process mining inspired algorithm to be used to preprocess data in intrusion detection systems (IDS). The algorithm is designed to process the network packet data and it works well in online mode for online intrusion detection. To test our algorithm, we used the CSE-CIC-IDS2018 dataset which contains several common attacks. The packet data was preprocessed with this algorithm and then fed into the detectors. We report on the experiments using the algorithm with different machine learning (M…
In this paper, we propose a new practical association rule mining algorithm for anomaly detection in Intrusion Detection System (IDS). First, with a view of anomaly cases being relatively rarely occurred in network packet database, we define a rare association rule among infrequent itemsets rather than the traditional association rule mining method. And then, we discuss an interest measure to catch differences between interesting relations and uninteresting ones, and what interest there is, and develop a hash based rare association rule mining algorithm for finding rare, but useful anomaly pat…
In intrusion detection systems, classifiers still suffer from several drawbacks such as data dimensionality and dominance, different network feature types, and data impact on the classification. In this paper two significant enhancements are presented to solve these drawbacks. The first enhancement is an improved feature selection using sequential backward search and information gain. This, in turn, extracts valuable features that enhance positively the detection rate and reduce the false positive rate. The second enhancement is transferring nominal network features to numeric ones by exploiti…
Many modern intrusion detection systems are based on data mining and database-centric architecture, where a number of data mining techniques have been found. Among the most popular techniques, association rule mining is one of the important topics in data mining research. This approach determines interesting relationships between large sets of data items. This technique was initially applied to the so-called market basket analysis, which aims at finding regularities in shopping behaviour of customers of supermarkets. In contrast to dataset for market basket analysis, which takes usually hundre…
King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；Fayoum University（法尤姆大学）
Network intrusions have become a significant threat in recent years as a result of the increased demand of computer networks for critical systems. Intrusion detection system (IDS) has been widely deployed as a defense measure for computer networks. Features extracted from network traffic can be used as sign to detect anomalies. However with the huge amount of network traffic, collected data contains irrelevant and redundant features that affect the detection rate of the IDS, consumes high amount of system resources, and slowdown the training and testing process of the IDS. In this paper, a new…
Few-Shot Network Intrusion Detection Using Online Triplet Mining
Department of Electronics and Electrical Engineering, University of Strathclyde（斯特拉斯克莱德大学电子与电气工程系）；Faculty of Computer and Information Sciences, Ain Shams University（爱思曼大学计算机与信息科学学院）；Faculty of Electrical Engineering, Czech Technical University（捷克技术大学电气工程学院）
School of Electrical Engineering & Computer Science, University of North Dakota（北达科他大学电气工程与计算机科学学院）；National Institute of Posts and Telecommunication（国家邮电学院）
Smart grid is an emerging and promising technology. It uses the power of information technologies to deliver intelligently the electrical power to customers, and it allows the integration of the green technology to meet the environmental requirements. Unfortunately, information technologies have its inherent vulnerabilities and weaknesses that expose the smart grid to a wide variety of security risks. The Intrusion detection system (IDS) plays an important role in securing smart grid networks and detecting malicious activity, yet it suffers from several limitations. Many research papers have b…
Cluster-based Wireless Sensor Network (CWSN) is a kind of WSNs that because of avoiding long distance communications, preserve the energy of nodes and so is attractive for related applications. The criticality of most applications of WSNs and also their unattended nature, makes sensor nodes often susceptible to many types of attacks. Based on this fact, it is clear that cluster heads (CHs) are the most attacked targets by attackers, and also according to their critical operations in CWSNs, their compromise and control by an attacker will disrupt the entire cluster and sometimes the entire netw…
Security and distributed infrastructure are two of the most common requirements for big data software. But the security features of the big data platforms are still premature. It is critical to identify, modify, test and execute some of the existing security mechanisms before using them in the big data world. In this paper, we propose a novel intrusion detection technique that understands and works according to the needs of big data systems. Our proposed technique identifies program level anomalies using two methods - a profiling method that models application behavior by creating process sign…
University of the Basque Country (UPV-EHU)（巴斯克大学（UPV-EHU））
The identification of cyberattacks which target information and communication systems has been a focus of the research community for years. Network intrusion detection is a complex problem which presents a diverse number of challenges. Many attacks currently remain undetected, while newer ones emerge due to the proliferation of connected devices and the evolution of communication technology. In this survey, we review the methods that have been applied to network data with the purpose of developing an intrusion detector, but contrary to previous reviews in the area, we analyze them from the per…
York University（约克大学）；iSecurity Consulting Inc（iSecurity咨询公司）；Dapasoft Inc（Dapasoft公司）
Industrial Information Technology (IT) infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This paper aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detec…
The research literature on cybersecurity incident detection & response is very rich in automatic detection methodologies, in particular those based on the anomaly detection paradigm. However, very little attention has been devoted to the diagnosis ability of the methods, aimed to provide useful information on the causes of a given detected anomaly. This information is of utmost importance for the security team to reduce the time from detection to response. In this paper, we present Multivariate Big Data Analysis (MBDA), a complete intrusion detection approach based on 5 steps to effectively ha…
Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction
International University of Business Agriculture and Technology（国际商业农业技术大学）；Jagannath University（贾甘纳特大学）；Deakin University（迪肯大学）；University of New South Wales（新南威尔士大学）；Imam Mohammad Ibn Saud Islamic University（伊玛目穆罕默德·伊本·沙特伊斯兰大学）；The University of Queensland（昆士兰大学）
Huge datasets in cyber security, such as network traffic logs, can be analyzed using machine learning and data mining methods. However, the amount of collected data is increasing, which makes analysis more difficult. Many machine learning methods have not been designed for big datasets, and consequently are slow and difficult to understand. We address the issue of efficient network traffic classification by creating an intrusion detection framework that applies dimensionality reduction and conjunctive rule extraction. The system can perform unsupervised anomaly detection and use this informati…
With the growing use of information technology in all life domains, hacking has become more negatively effective than ever before. Also with developing technologies, attacks numbers are growing exponentially every few months and become more sophisticated so that traditional IDS becomes inefficient detecting them. This paper proposes a solution to detect not only new threats with higher detection rate and lower false positive than already used IDS, but also it could detect collective and contextual security attacks. We achieve those results by using Networking Chatbot, a deep recurrent neural n…
Recently, there has been an interest in improving the resources available in Intrusion Detection System (IDS) techniques. In this sense, several studies related to cybersecurity show that the environment invasions and information kidnapping are increasingly recurrent and complex. The criticality of the business involving operations in an environment using computing resources does not allow the vulnerability of the information. Cybersecurity has taken on a dimension within the universe of indispensable technology in corporations, and the prevention of risks of invasions into the environment is …
Following the current big data trend, the scale of real-time system call traces generated by Linux applications in a contemporary data center may increase excessively. Due to the deficiency of scalability, it is challenging for traditional host-based intrusion detection systems deployed on every single host to collect, maintain, and manipulate those large-scale accumulated system call traces. It is inflexible to build data mining models on one physical host that has static computing capability and limited storage capacity. To address this issue, we propose SCADS, a corresponding solution using…
Intrusion detection systems (IDSs) have become a widely used measure for security systems. The main problem for those systems results is the irrelevant alerts on those results. We will propose a data mining based method for classification to distinguish serious alerts and irrelevant one with a performance of 99.9% which is better in comparison with the other recent data mining methods that have reached the performance of 97%. A ranked alerts list also created according to alerts importance to minimize human interventions.
Vellore Institute of Technology（韦洛尔理工大学）；PSG College of Technology（PSG理工学院）
Sparse events, such as malign attacks in real-time network traffic, have caused big organisations an immense hike in revenue loss. This is due to the excessive growth of the network and its exposure to a plethora of people. The standard methods used to detect intrusions are not promising and have significant failure to identify new malware. Moreover, the challenges in handling high volume data with sparsity, high false positives, fewer detection rates in minor class, training time and feature engineering of the dimensionality of data has promoted deep learning to take over the task with less t…
Birla Institute of Technology, Mesra, Ranchi（比尔拉理工学院，梅斯拉，兰契）
Recent advances in technology have made our work easier compare to earlier times. Computer network is growing day by day but while discussing about the security of computers and networks it has always been a major concerns for organizations varying from smaller to larger enterprises. It is true that organizations are aware of the possible threats and attacks so they always prepare for the safer side but due to some loopholes attackers are able to make attacks. Intrusion detection is one of the major fields of research and researchers are trying to find new algorithms for detecting intrusions. …
In this paper, we introduce new learning algorithms for reducing false positives in intrusion detection. It is based on decision tree-based attribute weighting with adaptive naïve Bayesian tree, which not only reduce the false positives (FP) at acceptable level, but also scale up the detection rates (DR) for different types of network intrusions. Due to the tremendous growth of network-based services, intrusion detection has emerged as an important technique for network security. Recently data mining algorithms are applied on network-based traffic data and host-based program behaviors to detec…
Network intrusion detection is one of the most visible uses for Big Data analytics. One of the main problems in this application is the constant rise of new attacks. This scenario, characterized by the fact that not enough labeled examples are available for the new classes of attacks is hardly addressed by traditional machine learning approaches. New findings on the capabilities of Zero-Shot learning (ZSL) approach makes it an interesting solution for this problem because it has the ability to classify instances of unseen classes. ZSL has inherently two stages: the attribute learning and the i…
School of Computing and Analytics, NSHM Knowledge Campus（NSHM知识园区计算与分析学院）
Machine learning and data mining algorithms play important roles in designing intrusion detection systems. Based on their approaches toward the detection of attacks in a network, intrusion detection systems can be broadly categorized into two types. In the misuse detection systems, an attack in a system is detected whenever the sequence of activities in the network matches with a known attack signature. In the anomaly detection approach, on the other hand, anomalous states in a system are identified based on a significant difference in the state transitions of the system from its normal states…
In this era of internet, E-Business and e-commerce applications are using Databases as their integral part. These Databases irrespective of the technology used are vulnerable to SQL injection attacks. These Attacks are considered very dangerous as well as very easy to use for attackers and intruders. In this paper, we are proposing a new approach to detect intrusion from attackers by using SQL injection. The main idea of our proposed solution is to create trusted user profiles fetched from the Queries submitted by authorized users by using association rules. After that we will use a hybrid (an…
Prince Al-Hussein Bin Abdullah II Faculty of Information Technology, Hashemite University（哈希姆大学侯赛因·本·阿卜杜拉二世亲王信息技术学院）；Princess Sumaya University for Technology（苏玛雅公主科技大学）；University of Miskolc（米什科尔茨大学）
Network security engineers work to keep services available all the time by handling intruder attacks. Intrusion Detection System (IDS) is one of the obtainable mechanism that used to sense and classify any abnormal actions. Therefore, the IDS must be always up to date with the latest intruder attacks signatures to preserve confidentiality, integrity and availability of the services. The speed of the IDS is very important issue as well learning the new attacks. This research work illustrates how the Knowledge Discovery and Data Mining (or Knowledge Discovery in Databases) KDD dataset is very ha…
University of Lahore（拉合尔大学）；NUCES（巴基斯坦国家计算机与新兴科学大学）
Network security has become the biggest concern in the area of cyber security because of the exponential growth in computer networks and applications. Intrusion detection plays an important role in the security of information systems or networks devices. The purpose of an intrusion detection system (IDS) is to detect malicious activities and then generate an alarm against these activities. Having a large amount of data is one of the key problems in detecting attacks. Most of the intrusion detection systems use all features of datasets to evaluate the models and result in is, low detection rate…
Network intrusion detection systems are an active area of research to identify threats that face computer networks. Network packets comprise of high dimensions which require huge effort to be examined effectively. As these dimensions contain some irrelevant features, they cause a high False Alarm Rate (FAR). In this paper, we propose a hybrid method as a feature selection, based on the central points of attribute values and an Association Rule Mining algorithm to decrease the FAR. This algorithm is designed to be implemented in a short processing time, due to its dependency on the central poin…
School of Mathematics and Computer Science, University of Yachay Tech（亚奇理工大学数学与计算机科学学院）
Intrusion detection into computer networks has become one of the most important issues in cybersecurity. Attackers keep on researching and coding to discover new vulnerabilities to penetrate information security system. In consequence computer systems must be daily upgraded using up-to-date techniques to keep hackers at bay. This paper focuses on the design and implementation of an intrusion detection system based on Deep Learning architectures. As a first step, a shallow network is trained with labelled log-in [into a computer network] data taken from the Dataset CICIDS2017. The internal beha…
Network security engineers work to keep services available all the time by handling intruder attacks. Intrusion Detection System (IDS) is one of the obtainable mechanisms that is used to sense and classify any abnormal actions. Therefore, the IDS must be always up to date with the latest intruder attacks signatures to preserve confidentiality, integrity, and availability of the services. The speed of the IDS is a very important issue as well learning the new attacks. This research work illustrates how the Knowledge Discovery and Data Mining (or Knowledge Discovery in Databases) KDD dataset is …
Artificial Intelligence (AI) has become an integral part of modern-day security solutions for its ability to learn very complex functions and handling "Big Data". However, the lack of explainability and interpretability of successful AI models is a key stumbling block when trust in a model's prediction is critical. This leads to human intervention, which in turn results in a delayed response or decision. While there have been major advancements in the speed and performance of AI-based intrusion detection systems, the response is still at human speed when it comes to explaining and interpreting…
Computing technologies pervade physical spaces and human lives, and produce a vast amount of data that is available for analysis. However, there is a growing concern that potentially sensitive data may become public if the collected data are not appropriately sanitized before being released for investigation. Although there are more than a few privacy-preserving methods available, they are not efficient, scalable or have problems with data utility, and/or privacy. This paper addresses these issues by proposing an efficient and scalable nonreversible perturbation algorithm, PABIDOT, for privacy…
The internet has caused tremendous changes since its appearance in the 1980s, and now, the Internet of Things (IoT) seems to be doing the same. The potential of IoT has made it the center of attention for many people, but, where some see an opportunity to contribute, others may see IoT networks as a target to be exploited. The high number of IoT devices makes them the perfect setup for staging denial-of-service attacks (DoS) that can have devastating consequences. This renders the need for cybersecurity measures such as intrusion detection systems (IDSs) evident. The aim of this paper is to bu…
CST-AFNet: A dual attention-based deep learning framework for intrusion detection in IoT networks
Skill Morph；Bangladesh Army University of Science & Technology（孟加拉国陆军科技大学）；Daffodil International University（达福迪尔国际大学）；State University of Bangladesh（孟加拉国州立大学）
Network traffic analytics technology is a cornerstone for cyber security systems. We demonstrate its use through three popular and contemporary cyber security applications in intrusion detection, malware analysis and botnet detection. However, automated traffic analytics faces the challenges raised by big traffic data. In terms of big data's three characteristics --- volume, variety and velocity, we review three state of the art techniques to mitigate the key challenges including real-time traffic classification, unknown traffic classification, and efficiency of classifiers. The new techniques…
Big data platforms such as Hadoop and Spark are being widely adopted both by academia and industry. In this paper, we propose a runtime intrusion detection technique that understands and works according to the properties of such distributed compute platforms. The proposed method is based on runtime analysis of system and library calls and memory access patterns of tasks running on the datanodes (slaves). First, the primary datanode of a big data system creates a behavior profile for every task it executes. A behavior profile includes (a) trace of the system & library calls made, and (b) sequen…
University of Technology Sydney（悉尼科技大学）；Tsinghua University（清华大学）
Data mining is a key technology in big data analytics and it can discover understandable knowledge (patterns) hidden in large data sets. Association rule is one of the most useful knowledge patterns, and a large number of algorithms have been developed in the data mining literature to generate association rules corresponding to different problems and situations. Privacy becomes a vital issue when data mining is used to sensitive data sets like medical records, commercial data sets and national security. In this Letter, we present a quantum protocol for mining association rules on vertically pa…
There are hardly any data sets publicly available that can be used to evaluate intrusion detection algorithms. The biggest threat for industrial applications arises from state-sponsored and criminal groups. Often, formerly unknown exploits are employed by these attackers, so-called 0-day exploits. They cannot be discovered with signature-based intrusion detection. Thus, statistical or machine learning based anomaly detection lends itself readily. These methods especially, however, need a large amount of labelled training data. In this work, an exemplary industrial use case with real-world indu…
Attackers have developed ever more sophisticated and intelligent ways to hack information and communication technology systems. The extent of damage an individual hacker can carry out upon infiltrating a system is well understood. A potentially catastrophic scenario can be envisaged where a nation-state intercepting encrypted financial data gets hacked. Thus, intelligent cybersecurity systems have become inevitably important for improved protection against malicious threats. However, as malware attacks continue to dramatically increase in volume and complexity, it has become ever more challeng…
The network security analyzers use intrusion detection systems (IDSes) to distinguish malicious traffic from benign ones. The deep learning-based IDSes are proposed to auto-extract high-level features and eliminate the time-consuming and costly signature extraction process. However, this new generation of IDSes still suffers from a number of challenges. One of the main issues of an IDS is facing traffic concept drift which manifests itself as new (i.e., zero-day) attacks, in addition to the changing behavior of benign users/applications. Furthermore, a practical DL-based IDS needs to be confor…
Modern networked systems are constantly under threat from systemic attacks. There has been a massive upsurge in the number of devices connected to a network as well as the associated traffic volume. This has intensified the need to better understand all possible attack vectors during system design and implementation. Further, it has increased the need to mine large data sets, analyzing which has become a daunting task. It is critical to scale monitoring infrastructures to match this need, but a difficult goal for the small and medium organization. Hence, there is a need to propose novel approa…
Secure Multi-Party Computation (SMC) allows parties with similar background to compute results upon their private data, minimizing the threat of disclosure. The exponential increase in sensitive data that needs to be passed upon networked computers and the stupendous growth of internet has precipitated vast opportunities for cooperative computation, where parties come together to facilitate computations and draw out conclusions that are mutually beneficial; at the same time aspiring to keep their private data secure. These computations are generally required to be done between competitors, who…
Recent years have seen the rise of more sophisticated attacks including advanced persistent threats (APTs) which pose severe risks to organizations and governments by targeting confidential proprietary information. Additionally, new malware strains are appearing at a higher rate than ever before. Since many of these malware are designed to evade existing security products, traditional defenses deployed by most enterprises today, e.g., anti-virus, firewalls, intrusion detection systems, often fail at detecting infections at an early stage. We address the problem of detecting early-stage infecti…
In the current world, the Internet is being used almost everywhere. With the rise of IoT technology, which is one of the most used technologies, billions of IoT devices are interconnected over the Internet. However, DoS/DDoS attacks are the most frequent and perilous threat to this growing technology. New types of DDoS attacks are highly advanced and complicated, and it is almost impossible to detect or mitigate by the existing intrusion detection systems and traditional methods. Fortunately, Big Data, Data mining, and Machine Learning technologies make it possible to detect DDoS traffic effec…
The issue of increasing volume, variety and velocity of has been an area of concern in cloud forensics. The high volume of data will, at some point, become computationally exhaustive to be fully extracted and analysed in a timely manner. To cut down the size of investigation, it is important for a digital forensic practitioner to possess a well-rounded knowledge about the most relevant data artefacts from the cloud product investigating. In this paper, we seek to tackle on the residual artefacts from the use of CloudMe cloud storage service. We demonstrate the types and locations of the artefa…
