In modern highly interconnected power grids, automatic generation control (AGC) is crucial in maintaining the stability of the power grid. The dependence of the AGC system on the information and communications technology (ICT) system makes it vulnerable to various types of cyber-attacks. Thus, information flow (IF) analysis and anomaly detection became paramount for preventing cyber attackers from driving the cyber-physical power system (CPPS) to instability. In this paper, the ICT network traffic rules in CPPSs are explored and the frequency domain features of the ICT network traffic are extracted, basically for developing a robust learning algorithm that can learn the normal traffic pattern based on the ResNeSt convolutional neural network (CNN). Furthermore, to overcome the problem of insufficient abnormal traffic labeled samples, transfer learning approach is used. In the proposed data-driven-based method the deep learning model is trained by traffic frequency features, which makes our model robust against AGC's parameters uncertainties and modeling nonlinearities.
DDoS attacks are simple, effective, and still pose a significant threat even after more than two decades. Given the recent success in machine learning, it is interesting to investigate how we can leverage deep learning to filter out application layer attack requests. There are challenges in adopting deep learning solutions due to the ever-changing profiles, the lack of labeled data, and constraints in the online setting. Offline unsupervised learning methods can sidestep these hurdles by learning an anomaly detector $N$ from the normal-day traffic ${\mathcal N}$. However, anomaly detection doe…
Know Unreported Roadway Incidents in Real-time: Early Traffic Anomaly Detection
Department of Civil and Environmental Engineering, Carnegie Mellon University（土木与环境工程系，卡内基梅隆大学）；Heinz College, Carnegie Mellon University（海因兹学院，卡内基梅隆大学）
Event detection has been an important task in transportation, whose task is to detect points in time when large events disrupts a large portion of the urban traffic network. Travel information {Origin-Destination} (OD) matrix data by map service vendors has large potential to give us insights to discover historic patterns and distinguish anomalies. However, to fully capture the spatial and temporal traffic patterns remains a challenge, yet serves a crucial role for effective anomaly detection. Meanwhile, existing anomaly detection methods have not well-addressed the extreme data sparsity and h…
Network Anomaly Traffic Detection via Multi-view Feature Fusion
Institute of Cyberspace Security, Zhejiang University of Technology（浙江工业大学网络空间安全研究院）；Binjiang Institute of Artificial Intelligence, Zhejiang University of Technology（杭州市滨江区浙工大人工智能创新研究院）
Identifying anomaly multimedia traffic in cyberspace is a big challenge in distributed service systems, multiple generation networks and future internet of everything. This letter explores meta-generalization for a multiparty privacy learning model in graynet to improve the performance of anomaly multimedia traffic identification. The multiparty privacy learning model in graynet is a globally shared model that is partitioned, distributed and trained by exchanging multiparty parameters updates with preserving private data. The meta-generalization refers to discovering the inherent attributes of…
University of Science and Technology Beijing（北京科技大学）；COMSATS University Islamabad（COMSATS伊斯兰堡大学）
Mobile networks possess information about the users as well as the network. Such information is useful for making the network end-to-end visible and intelligent. Big data analytics can efficiently analyze user and network information, unearth meaningful insights with the help of machine learning tools. Utilizing big data analytics and machine learning, this work contributes in three ways. First, we utilize the call detail records (CDR) data to detect anomalies in the network. For authentication and verification of anomalies, we use k-means clustering, an unsupervised machine learning algorithm…
University of Sydney（悉尼大学）；NICTA（澳大利亚国家信息与通信技术研究院）
In this paper we focus on the detection of network anomalies like Denial of Service (DoS) attacks and port scans in a unified manner. While there has been an extensive amount of research in network anomaly detection, current state of the art methods are only able to detect one class of anomalies at the cost of others. The key tool we will use is based on the spectral decomposition of a trajectory/hankel matrix which is able to detect deviations from both between and within correlation present in the observed network traffic data. Detailed experiments on synthetic and real network traces shows …
CNN+Transformer Based Anomaly Traffic Detection in UAV Networks for Emergency Rescue
The Key Laboratory of Dynamic Cognitive System of Electromagnetic Spectrum Space, Ministry of Industry and Information Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学电磁频谱空间动态认知系统重点实验室（工业和信息化部））；Department of Electronic and Electrical Engineering, University College London（伦敦大学学院电子与电气工程系）
The monitoring and management of high-volume feature-rich traffic in large networks offers significant challenges in storage, transmission and computational costs. The predominant approach to reducing these costs is based on performing a linear mapping of the data to a low-dimensional subspace such that a certain large percentage of the variance in the data is preserved in the low-dimensional representation. This variance-based subspace approach to dimensionality reduction forces a fixed choice of the number of dimensions, is not responsive to real-time shifts in observed traffic patterns, and…
Evaluating Temporal and Structural Anomaly Detection Paradigms for DDoS Traffic
Institute of Exact and Technological Sciences（精确与技术科学研究所）；Federal University of Viçosa（弗雷塔斯联邦大学）；Department of Informatics -- School of Engineering（信息学院 -- 工程学院）；University of Minho（明霍大学）；University of São Paulo（圣保罗大学）
University of Technology Sydney（悉尼科技大学）；Australian National University（澳大利亚国立大学）
Congestion prediction represents a major priority for traffic management centres around the world to ensure timely incident response handling. The increasing amounts of generated traffic data have been used to train machine learning predictors for traffic, however, this is a challenging task due to inter-dependencies of traffic flow both in time and space. Recently, deep learning techniques have shown significant prediction improvements over traditional models, however, open questions remain around their applicability, accuracy and parameter tuning. This paper brings two contributions in terms…
We evaluate methods for applying unsupervised anomaly detection to cybersecurity applications on computer network traffic data, or flow. We borrow from the natural language processing literature and conceptualize flow as a sort of "language" spoken between machines. Five sequence aggregation rules are evaluated for their efficacy in flagging multiple attack types in a labeled flow dataset, CICIDS2017. For sequence modeling, we rely on long short-term memory (LSTM) recurrent neural networks (RNN). Additionally, a simple frequency-based model is described and its performance with respect to atta…
Institute of Command Information System, PLA University of Science and Technology（中国人民解放军理工大学指挥信息系统学院）
Network anomaly detection is still a vibrant research area. As the fast growth of network bandwidth and the tremendous traffic on the network, there arises an extremely challengeable question: How to efficiently and accurately detect the anomaly on multiple traffic? In multi-task learning, the traffic consisting of flows at different time periods is considered as a task. Multiple tasks at different time periods performed simultaneously to detect anomalies. In this paper, we apply the multi-task feature selection in network anomaly detection area which provides a powerful method to gather infor…
The detection of contextual anomalies is a challenging task for surveillance since an observation can be considered anomalous or normal in a specific environmental context. An unmanned aerial vehicle (UAV) can utilize its aerial monitoring capability and employ multiple sensors to gather contextual information about the environment and perform contextual anomaly detection. In this work, we introduce a deep neural network-based method (CADNet) to find point anomalies (i.e., single instance anomalous data) and contextual anomalies (i.e., context-specific abnormality) in an environment using a UA…
Traffic congestion anomaly detection is of paramount importance in intelligent traffic systems. The goals of transportation agencies are two-fold: to monitor the general traffic conditions in the area of interest and to locate road segments under abnormal congestion states. Modeling congestion patterns can achieve these goals for citywide roadways, which amounts to learning the distribution of multivariate time series (MTS). However, existing works are either not scalable or unable to capture the spatial-temporal information in MTS simultaneously. To this end, we propose a principled and compr…
Using a convGRU-based autoencoder, this thesis proposes a framework to learn spatial-temporal aspects of raw network traffic in an unsupervised and protocol-agnostic manner. The learned representations are used to measure the effect on the results of a subsequent anomaly detection and are compared to the application without the extracted features. The evaluation showed, that the anomaly detection could not effectively be enhanced when applied on compressed traffic fragments for the context of network intrusion detection. Yet, the trained autoencoder successfully generates a compressed represen…
Over the last two decades, a lot of work has been done in improving network security, particularly in intrusion detection systems (IDS) and anomaly detection. Machine learning solutions have also been employed in IDSs to detect known and plausible attacks in incoming traffic. Parameters such as packet contents, sender IP and sender port, connection duration, etc. have been previously used to train these machine learning models to learn to differentiate genuine traffic from malicious ones. Generative Adversarial Networks (GANs) have been significantly successful in detecting such anomalies, mos…
Spatiotemporal traffic data (e.g., link speed/flow) collected from sensor networks can be organized as multivariate time series with additional spatial attributes. A crucial task in analyzing such data is to identify and detect anomalous observations and events from the data with complex spatial and temporal dependencies. Robust Principal Component Analysis (RPCA) is a widely used tool for anomaly detection. However, the traditional RPCA purely relies on the global low-rank assumption while ignoring the local temporal correlations. In light of this, this study proposes a Hankel-structured tens…
NUCES-FAST（巴基斯坦国立计算机与新兴科学大学）；University of Gloucestershire（格洛斯特郡大学）
Anomaly detection is a crucial step for preventing malicious activities in the network and keeping resources available all the time for legitimate users. It is noticed from various studies that classical anomaly detectors work well with small and sampled data, but the chances of failures increase with real-time (non-sampled data) traffic data. In this paper, we will be exploring security analytic techniques for DDoS anomaly detection using different machine learning techniques. In this paper, we are proposing a novel approach which deals with real traffic as input to the system. Further, we st…
Huge datasets in cyber security, such as network traffic logs, can be analyzed using machine learning and data mining methods. However, the amount of collected data is increasing, which makes analysis more difficult. Many machine learning methods have not been designed for big datasets, and consequently are slow and difficult to understand. We address the issue of efficient network traffic classification by creating an intrusion detection framework that applies dimensionality reduction and conjunctive rule extraction. The system can perform unsupervised anomaly detection and use this informati…
Although home IoT (Internet of Things) devices are typically plain and task oriented, the context of their daily use may affect their traffic patterns. For this reason, anomaly-based intrusion detection systems tend to suffer from a high false positive rate (FPR). To overcome this, we propose a two-step collaborative anomaly detection method which first uses an autoencoder to differentiate frequent (`benign') and infrequent (possibly `malicious') traffic flows. Clustering is then used to analyze only the infrequent flows and classify them as either known ('rare yet benign') or unknown (`malici…
As the number of heterogenous IP-connected devices and traffic volume increase, so does the potential for security breaches. The undetected exploitation of these breaches can bring severe cybersecurity and privacy risks. Anomaly-based \acp{IDS} play an essential role in network security. In this paper, we present a practical unsupervised anomaly-based deep learning detection system called ARCADE (Adversarially Regularized Convolutional Autoencoder for unsupervised network anomaly DEtection). With a convolutional \ac{AE}, ARCADE automatically builds a profile of the normal traffic using a subse…
Explainable Autoencoder-Based Anomaly Detection in IEC 61850 GOOSE Networks
Universidad Rey Juan Carlos（雷奥恩卡洛斯大学）；Red Eléctrica (REDEIA Group)（电力公司（REDEIA小组））；Centro de Investigación for Data, Complex Networks and Cybersecurity Sciences（数据、复杂网络和网络安全科学研究中心）
Pohang University of Science and Technology（浦项科技大学）
Anomaly detection in database management systems (DBMSs) is difficult because of increasing number of statistics (stat) and event metrics in big data system. In this paper, I propose an automatic DBMS diagnosis system that detects anomaly periods with abnormal DB stat metrics and finds causal events in the periods. Reconstruction error from deep autoencoder and statistical process control approach are applied to detect time period with anomalies. Related events are found using time series similarity measures between events and abnormal stat metrics. After training deep autoencoder with DBMS me…
Ho Chi Minh City University of Technology, VNUHCM（越南胡志明市理工大学（越南国家大学胡志明市分校））；Ho Chi Minh City University of Technology and Education（胡志明市工业大学）
Through continuous observation and modeling of normal behavior in networks, Anomaly-based Network Intrusion Detection System (A-NIDS) offers a way to find possible threats via deviation from the normal model. The analysis of network traffic based on the time series model has the advantage of exploiting the relationship between packages within network traffic and observing trends of behaviors over a period of time. It will generate new sequences with good features that support anomaly detection in network traffic and provide the ability to detect new attacks. Besides, an anomaly detection techn…
Protecting the networks of tomorrow is set to be a challenging domain due to increasing cyber security threats and widening attack surfaces created by the Internet of Things (IoT), increased network heterogeneity, increased use of virtualisation technologies and distributed architectures. This paper proposes SDS (Software Defined Security) as a means to provide an automated, flexible and scalable network defence system. SDS will harness current advances in machine learning to design a CNN (Convolutional Neural Network) using NAS (Neural Architecture Search) to detect anomalous network traffic.…
Norwegian University of Science and Technology（挪威科技大学）
Over the past decade, several approaches have been introduced for short-term traffic prediction. However, providing fine-grained traffic prediction for large-scale transportation networks where numerous detectors are geographically deployed to collect traffic data is still an open issue. To address this issue, in this paper, we formulate the problem of customizing an LSTM model for a single detector into a finite Markov decision process and then introduce an Automatic LSTM Customization (ALC) algorithm to automatically customize an LSTM model for a single detector such that the corresponding p…
MamNet: A Novel Hybrid Model for Time-Series Forecasting and Frequency Pattern Analysis in Network Traffic
Columbia University（哥伦比亚大学）；University of California, Irvine（加利福尼亚大学欧文分校）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Southern California（南加利福尼亚大学）；Meta Platform Inc.（Meta平台公司）；Northeastern University（东北大学）
We continue to develop our neural network (NN) based forecasting approach to anomaly detection (AD) using the Secure Water Treatment (SWaT) industrial control system (ICS) testbed dataset. We propose genetic algorithms (GA) to find the best NN architecture for a given dataset, using the NAB metric to assess the quality of different architectures. The drawbacks of the F1-metric are analyzed. Several techniques are proposed to improve the quality of AD: exponentially weighted smoothing, mean p-powered error measure, individual error weight for each variable, disjoint prediction windows. Based on…
Cyberthreats are a permanent concern in our modern technological world. In the recent years, sophisticated traffic analysis techniques and anomaly detection (AD) algorithms have been employed to face the more and more subversive adversarial attacks. A malicious intrusion, defined as an invasive action intending to illegally exploit private resources, manifests through unusual data traffic and/or abnormal connectivity pattern. Despite the plethora of statistical or signature-based detectors currently provided in the literature, the topological connectivity component of a malicious flow is less …
Hong Kong Baptist University（香港浸会大学）；The University of Hong Kong（香港大学）
Modern cities experience heavy traffic flows and congestions regularly across space and time. Monitoring traffic situations becomes an important challenge for the Traffic Control and Surveillance Systems (TCSS). In advanced TCSS, it is helpful to automatically detect and classify different traffic incidents such as severity of congestion, abnormal driving pattern, abrupt or illegal stop on road, etc. Although most TCSS are equipped with basic incident detection algorithms, they are however crude to be really useful as an automated tool for further classification. In literature, there is a lack…
Xidian University（西安电子科技大学）；Pazhou Lab（琶洲实验室）；University of South Florida（南佛罗里达大学）
With the fast growing demand on new services and applications as well as the increasing awareness of data protection, traditional centralized traffic classification approaches are facing unprecedented challenges. This paper introduces a novel framework, Federated Generative Adversarial Networks and Automatic Classification (FGAN-AC), which integrates decentralized data synthesizing with traffic classification. FGAN-AC is able to synthesize and classify multiple types of service data traffic from decentralized local datasets without requiring a large volume of manually labeled dataset or causin…
Detecting faults in manufacturing applications can be difficult, especially if each fault model is to be engineered by hand. Data-driven approaches, using Machine Learning (ML) for detecting faults have recently gained increasing interest, where a ML model can be trained on a set of data from a manufacturing process. In this paper, we present a use case of using ML models for detecting faults during automated screwdriving operations, and introduce a new dataset containing fully monitored and registered data from a Universal Robot and OnRobot screwdriver during both normal and anomalous operati…
Sotera Defense Solutions, a KeyW Company（索特拉防务解决方案公司（KeyW旗下公司））；Cybraics（赛博艾克斯公司）；DZYNE Technologies（迪赞科技公司）；Cynnovative（辛诺瓦蒂公司）
We show that a recurrent neural network is able to learn a model to represent sequences of communications between computers on a network and can be used to identify outlier network traffic. Defending computer networks is a challenging problem and is typically addressed by manually identifying known malicious actor behavior and then specifying rules to recognize such behavior in network communications. However, these rule-based approaches often generalize poorly and identify only those patterns that are already known to researchers. An alternative approach that does not rely on known malicious …
York University（约克大学）；iSecurity Consulting Inc（iSecurity咨询公司）；Dapasoft Inc（Dapasoft公司）
Industrial Information Technology (IT) infrastructures are often vulnerable to cyberattacks. To ensure security to the computer systems in an industrial environment, it is required to build effective intrusion detection systems to monitor the cyber-physical systems (e.g., computer networks) in the industry for malicious activities. This paper aims to build such intrusion detection systems to protect the computer networks from cyberattacks. More specifically, we propose a novel unsupervised machine learning approach that combines the K-Means algorithm with the Isolation Forest for anomaly detec…
University of Pisa（比萨大学）；CNR, Institute of Informatics and Telematics（意大利国家研究委员会信息学与电信研究所）；CNR, Institute of Information Science and Technologies（意大利国家研究委员会信息科学与技术研究所）
Automatic traffic classification is increasingly becoming important in traffic engineering, as the current trend of encrypting transport information (e.g., behind HTTP-encrypted tunnels) prevents intermediate nodes from accessing end-to-end packet headers. However, this information is crucial for traffic shaping, network slicing, and Quality of Service (QoS) management, for preventing network intrusion, and for anomaly detection. 3D networks offer multiple routes that can guarantee different levels of QoS. Therefore, service classification and separation are essential to guarantee the required…
A modification of the Random Forest algorithm for the categorization of traffic situations is introduced in this paper. The procedure yields an unsupervised machine learning method. The algorithm generates a proximity matrix which contains a similarity measure. This matrix is then reordered with hierarchical clustering to achieve a graphically interpretable representation. It is shown how the resulting proximity matrix can be visually interpreted and how the variation of the methods' metaparameter reveals different insights into the data. The proposed method is able to cluster data from any da…
In recent years, there have been many practical applications of anomaly detection such as in predictive maintenance, detection of credit fraud, network intrusion, and system failure. The goal of anomaly detection is to identify in the test data anomalous behaviors that are either rare or unseen in the training data. This is a common goal in predictive maintenance, which aims to forecast the imminent faults of an appliance given abundant samples of normal behaviors. Local outlier factor (LOF) is one of the state-of-the-art models used for anomaly detection, but the predictive performance of LOF…
Norwegian University of Science and Technology（挪威科技大学）；Simula Research Laboratory（西穆拉研究院）
Short-term traffic speed prediction has been an important research topic in the past decade, and many approaches have been introduced. However, providing fine-grained, accurate, and efficient traffic-speed prediction for large-scale transportation networks where numerous traffic detectors are deployed has not been well studied. In this paper, we propose DistPre, which is a distributed fine-grained traffic speed prediction scheme for large-scale transportation networks. To achieve fine-grained and accurate traffic-speed prediction, DistPre customizes a Long Short-Term Memory (LSTM) model with a…
