Monitoring network traffic data to detect any hidden patterns of anomalies is a challenging and time-consuming task that requires high computing resources. To this end, an appropriate summarization technique is of great importance, where it can be a substitute for the original data. However, the summarized data is under the threat of removing anomalies. Therefore, it is vital to create a summary that can reflect the same pattern as the original data. Therefore, in this paper, we propose an INtelligent Summarization approach for IDENTifying hidden anomalies, called INSIDENT. The proposed approach guarantees to keep the original data distribution in summarized data. Our approach is a clustering-based algorithm that dynamically maps original feature space to a new feature space by locally weighting features in each cluster. Therefore, in new feature space, similar samples are closer, and consequently, outliers are more detectable. Besides, selecting representatives based on cluster size keeps the same distribution as the original data in summarized data. INSIDENT can be used both as the preprocess approach before performing anomaly detection algorithms and anomaly detection algorithm. The experimental results on benchmark datasets prove a summary of the data can be a substitute for original data in the anomaly detection task.
Online Detection of Anomalies in Temporal Knowledge Graphs with Interpretability
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Department of Computer Science, Yale University（耶鲁大学计算机科学系）；Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China（电子科技大学深圳高等研究院）
Anomaly detection is important for industrial automation and part quality assurance, and while humans can easily detect anomalies in components given a few examples, designing a generic automated system that can perform at human or above human capabilities remains a challenge. In this work, we present a simple new anomaly detection algorithm called FADS (feature-based anomaly detection system) which leverages pretrained convolutional neural networks (CNN) to generate a statistical model of nominal inputs by observing the activation of the convolutional filters. During inference the system comp…
University of Science and Technology Beijing（北京科技大学）；COMSATS University Islamabad（COMSATS伊斯兰堡大学）
Mobile networks possess information about the users as well as the network. Such information is useful for making the network end-to-end visible and intelligent. Big data analytics can efficiently analyze user and network information, unearth meaningful insights with the help of machine learning tools. Utilizing big data analytics and machine learning, this work contributes in three ways. First, we utilize the call detail records (CDR) data to detect anomalies in the network. For authentication and verification of anomalies, we use k-means clustering, an unsupervised machine learning algorithm…
Deep Learning Approach to Anomaly Detection in Enterprise ETL Processes with Autoencoders
Cornell University（康奈尔大学）；The George Washington University（乔治华盛顿大学）；University of Southern California（南加州大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
NUCES-FAST（巴基斯坦国立计算机与新兴科学大学）；University of Gloucestershire（格洛斯特郡大学）
Anomaly detection is a crucial step for preventing malicious activities in the network and keeping resources available all the time for legitimate users. It is noticed from various studies that classical anomaly detectors work well with small and sampled data, but the chances of failures increase with real-time (non-sampled data) traffic data. In this paper, we will be exploring security analytic techniques for DDoS anomaly detection using different machine learning techniques. In this paper, we are proposing a novel approach which deals with real traffic as input to the system. Further, we st…
The rapid growth in stored time-oriented data necessitates the development of new methods for handling, processing, and interpreting large amounts of temporal data. One important example of such processing is detecting anomalies in time-oriented data. The Knowledge-Based Temporal Abstraction method was previously proposed for intelligent interpretation of temporal data based on predefined domain knowledge. In this study we propose a framework that integrates the KBTA method with a temporal pattern mining process for anomaly detection. According to the proposed method a temporal pattern mining …
An Optimal Cascade Feature-Level Spatiotemporal Fusion Strategy for Anomaly Detection in CAN Bus
School of Electrical and Computer Engineering, College of Engineering, University of Tehran（伊朗德黑兰技术大学电气与计算机工程学院）；Department of Electrical and Computer Engineering, Faculty of Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）
Traffic congestion anomaly detection is of paramount importance in intelligent traffic systems. The goals of transportation agencies are two-fold: to monitor the general traffic conditions in the area of interest and to locate road segments under abnormal congestion states. Modeling congestion patterns can achieve these goals for citywide roadways, which amounts to learning the distribution of multivariate time series (MTS). However, existing works are either not scalable or unable to capture the spatial-temporal information in MTS simultaneously. To this end, we propose a principled and compr…
Capital Normal University（首都师范大学）；Beijing Hospital（北京医院）；National Center of Gerontology（国家老年医学中心）；Institute of Geriatric Medicine, Chinese Academy of Medical Sciences（中国医学科学院老年医学研究所）；Savaid Medical School, University of Chinese Academy of Sciences（中国科学院大学存济医学院）；Beijing University of Technology（北京工业大学）；University of Chinese Academy of Sciences（中国科学院大学）
Artificial intelligence (AI) has brought tremendous impacts on biomedical sciences from academic researches to clinical applications, such as in biomarkers' detection and diagnosis, optimization of treatment, and identification of new therapeutic targets in drug discovery. However, the contemporary AI technologies, particularly deep machine learning (ML), severely suffer from non-interpretability, which might uncontrollably lead to incorrect predictions. Interpretability is particularly crucial to ML for clinical diagnosis as the consumers must gain necessary sense of security and trust from f…
A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval
A.K.Choudhury School of IT（A.K.Choudhury信息科技学院）；University of Calcutta（加尔各答大学）；Department of Computer Science（计算机科学系）；Bangabasi Morning College（Bangabasi晨学院）
As machine learning and cybersecurity continue to explode in the context of the digital ecosystem, the complexity of cybersecurity data combined with complicated and evasive machine learning algorithms leads to vast difficulties in designing an end to end system for intelligent, automatic anomaly classification. On the other hand, traditional systems use elementary statistics techniques and are often inaccurate, leading to weak centralized data analysis platforms. In this paper, we propose a novel system that addresses these two problems, titled CAMLPAD, for Cybersecurity Autonomous Machine Le…
This paper proposes a simple method to extract from a set of multiple related time series a compressed representation for each time series based on statistics for the entire set of all time series. This is achieved by a hierarchical algorithm that first generates an alphabet of shapelets based on the segmentation of centroids for clustered data, before labels of these shapelets are assigned to the segmentation of each single time series via nearest neighbor search using unconstrained dynamic time warping as distance measure to deal with non-uniform time series lenghts. Thereby, a sequence of l…
Wroclaw University of Science and Technology（弗罗茨瓦夫理工大学）
Hypertext transfer protocol (HTTP) is one of the most widely used protocols on the Internet. As a consequence, most attacks (i.e., SQL injection, XSS) use HTTP as the transport mechanism. Therefore, it is crucial to develop an intelligent solution that would allow to effectively detect and filter out anomalies in HTTP traffic. Currently, most of the anomaly detection systems are either rule-based or trained using manually selected features. We propose utilizing modern unsupervised language representation model for embedding HTTP requests and then using it to classify anomalies in the traffic. …
University of Bridgeport（桥港大学）；University of the Potomac（波托马克大学）；Westcliff University（韦斯特克利夫大学）；Gannon University（甘农大学）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；Washington University of Science and Technology（华盛顿科学技术大学）；Trine University（特莱恩大学）
Anomaly detection is concerned with identifying data patterns that deviate remarkably from the expected behaviour. This is an important research problem, due to its broad set of application domains, from data analysis to e-health, cybersecurity, predictive maintenance, fault prevention, and industrial automation. Herein, we review state-of-the-art methods that may be employed to detect anomalies in the specific area of sensor systems, which poses hard challenges in terms of information fusion, data volumes, data speed, and network/energy efficiency, to mention but the most pressing ones. In th…
An Attention-Based Deep Generative Model for Anomaly Detection in Industrial Control Systems
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Department of Computer Science, Universidad de las Fuerzas Armadas ESPE（厄瓜多尔武装部队大学计算机科学系）
Analysis of water and environmental data is an important aspect of many intelligent water and environmental system applications where inference from such analysis plays a significant role in decision making. Quite often these data that are collected through sensible sensors can be anomalous due to different reasons such as systems breakdown, malfunctioning of sensor detectors, and more. Regardless of their root causes, such data severely affect the results of the subsequent analysis. This paper demonstrates data cleaning and preparation for time-series data and further proposes cost-sensitive …
A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Software Technology, Zhejiang University（浙江大学软件技术学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）
The automatic dial of an emergency call - eCall - in response to a road accident is a feature that is gaining interest in the intelligent vehicle community. It indirectly increases the driving safety of road vehicles, but presents the technical challenge of developing an algorithm which triggers the emergency call only when needed, a non-trivial task for two-wheeled vehicles due to their complex dynamics. In the present work, we propose an eCall algorithm that detects these anomalies in the data time series, thanks to the cepstral analysis. The main advantage of the proposed approach is the di…
