Data representation plays a critical role in the performance of novelty detection (or ``anomaly detection'') methods in machine learning. The data representation of network traffic often determines the effectiveness of these models as much as the model itself. The wide range of novel events that network operators need to detect (e.g., attacks, malware, new applications, changes in traffic demands) introduces the possibility for a broad range of possible models and data representations. In each scenario, practitioners must spend significant effort extracting and engineering features that are most predictive for that situation or application. While anomaly detection is well-studied in computer networking, much existing work develops specific models that presume a particular representation -- often IPFIX/NetFlow. Yet, other representations may result in higher model accuracy, and the rise of programmable networks now makes it more practical to explore a broader range of representations. To facilitate such exploration, we develop a systematic framework, open-source toolkit, and public Python library that makes it both possible and easy to extract and generate features from network traffic and perform and end-to-end evaluation of these representations across most prevalent modern novelty detection models. We first develop and publicly release an open-source tool, an accompanying Python library (NetML), and end-to-end pipeline for novelty detection in network traffic. Second, we apply this tool to five different novelty detection problems in networking, across a range of scenarios from attack detection to novel device detection. Our findings general insights and guidelines concerning which features appear to be more appropriate for particular situations.
MamNet: A Novel Hybrid Model for Time-Series Forecasting and Frequency Pattern Analysis in Network Traffic
Columbia University（哥伦比亚大学）；University of California, Irvine（加利福尼亚大学欧文分校）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Southern California（南加利福尼亚大学）；Meta Platform Inc.（Meta平台公司）；Northeastern University（东北大学）
Anomaly Detection Framework Using Rule Extraction for Efficient Intrusion Detection
University of Jyväskylä（于韦斯屈莱大学）
网络安全中的庞大数据集，如网络流量日志，可以使用机器学习和数据采掘方法进行分析。然而，收集的数据量正在增加，这使得分析更加困难。许多机器学习方法不是为大数据集设计的，因此速度慢且难以理解。我们通过创建一个应用降维和合取规则提取的入侵检测框架来解决有效的网络流量分类问题。该系统可以执行无监督的异常检测，并使用此信息创建合取规则，对大量流量进行实时的分类。我们使用广泛使用的KDD Cup 99数据集和真实世界的网络日志对实现的系统进行了测试，以确认性能令人满意。这个系统是透明的，不像一个黑盒子，使它直观的领域专家，如网络管理员。
Columbia University（哥伦比亚大学）；University of Chicago（芝加哥大学）
Insecure Internet of things (IoT) devices pose significant threats to critical infrastructure and the Internet at large; detecting anomalous behavior from these devices remains of critical importance, but fast, efficient, accurate anomaly detection (also called "novelty detection") for these classes of devices remains elusive. One-Class Support Vector Machines (OCSVM) are one of the state-of-the-art approaches for novelty detection (or anomaly detection) in machine learning, due to their flexibility in fitting complex nonlinear boundaries between {normal} and {novel} data. IoT devices in smart…
In recent years, the rapid rise of video applications has led to an explosion of Internet video traffic, thereby posing severe challenges to network management. Therefore, effectively identifying and managing video traffic has become an urgent problem to be solved. However, the existing video traffic feature extraction methods mainly target at the traditional packet and flow level features, and the video traffic identification accuracy is low. Additionally, the issue of high data dimension often exists in video traffic identification, requiring an effective approach to select the most relevant…
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University（清华大学交叉信息研究院）
Traffic forecasting is crucial for public safety and resource optimization, yet is very challenging due to three aspects: i) current existing works mostly exploit intricate temporal patterns (e.g., the short-term thunderstorm and long-term daily trends) within a single method, which fail to accurately capture spatio-temporal dependencies under different schemas; ii) the under-exploration of the graph positional encoding limit the extraction of spatial information in the commonly used full graph attention network; iii) the quadratic complexity of the full graph attention introduces heavy comput…
School of ECE, College of Engineering, University of Tehran（德黑兰大学工程学院电气与计算机工程学院）
In this paper, we introduce a novel end-to-end traffic classification method to distinguish between traffic classes including VPN traffic in three layers of the Open Systems Interconnection (OSI) model. Classification of VPN traffic is not trivial using traditional classification approaches due to its encrypted nature. We utilize two well-known neural networks, namely multi-layer perceptron and recurrent neural network to create our cascade neural network focused on two metrics: class scores and distance from the center of the classes. Such approach combines extraction, selection, and classifi…
The intrusion detection system (IDS) is an essential element of security monitoring in computer networks. An IDS distinguishes the malicious traffic from the benign one and determines the attack types targeting the assets of the organization. The main challenge of an IDS is facing new (i.e., zero-day) attacks and separating them from benign traffic and existing types of attacks. Along with the power of the deep learning-based IDSes in auto-extracting high-level features and its independence from the time-consuming and costly signature extraction process, the mentioned challenge still exists in…
Université Savoie Mont Blanc（萨瓦勃朗峰大学）；Princeton University（普林斯顿大学）；Nokia Bell Labs（诺基亚贝尔实验室）；Inria（法国国家信息与自动化研究所）；University of Chicago（芝加哥大学）
Network management often relies on machine learning to make predictions about performance and security from network traffic. Often, the representation of the traffic is as important as the choice of the model. The features that the model relies on, and the representation of those features, ultimately determine model accuracy, as well as where and whether the model can be deployed in practice. Thus, the design and evaluation of these models ultimately requires understanding not only model accuracy but also the systems costs associated with deploying the model in an operational network. Towards …
The increasing success of Machine Learning (ML) and Deep Learning (DL) has recently re-sparked interest towards traffic classification. While classification of known traffic is a well investigated subject with supervised classification tools (such as ML and DL models) are known to provide satisfactory performance, detection of unknown (or zero-day) traffic is more challenging and typically handled by unsupervised techniques (such as clustering algorithms). In this paper, we share our experience on a commercial-grade DL traffic classification engine that is able to (i) identify known applicatio…
Monitoring network traffic to identify content, services, and applications is an active research topic in network traffic control systems. While modern firewalls provide the capability to decrypt packets, this is not appealing for privacy advocates. Hence, identifying any information from encrypted traffic is a challenging task. Nonetheless, previous work has identified machine learning methods that may enable application and service identification. The process involves high level feature extraction from network packet data then training a robust machine learning classifier for traffic identif…
Despite the use of machine learning for many network traffic analysis tasks in security, from application identification to intrusion detection, the aspects of the machine learning pipeline that ultimately determine the performance of the model -- feature selection and representation, model selection, and parameter tuning -- remain manual and painstaking. This paper presents a method to automate many aspects of traffic analysis, making it easier to apply machine learning techniques to a wider variety of traffic analysis tasks. We introduce nPrint, a tool that generates a unified packet represe…
Software Defined Networking (SDN) achieves programmability of a network through separation of the control and data planes. It enables flexibility in network management and control. Energy efficiency is one of the challenging global problems which has both economic and environmental impact. A massive amount of information is generated in the controller of an SDN based network. Machine learning gives the ability to computers to progressively learn from data without having to write specific instructions. In this work, we propose MER-SDN: a machine learning framework for traffic-aware energy effic…
Georgia Institute of Technology（佐治亚理工学院）；Trustwave (Singapore)（Trustwave（新加坡））；TUMCREATE (Singapore)（TUMCREATE（新加坡））；Heriot-Watt University（赫瑞-瓦特大学）；National University of Singapore（新加坡国立大学）
Classification of IoT devices into different types is of paramount importance, from multiple perspectives, including security and privacy aspects. Recent works have explored machine learning techniques for fingerprinting (or classifying) IoT devices, with promising results. However, existing works have assumed that the features used for building the machine learning models are readily available or can be easily extracted from the network traffic; in other words, they do not consider the costs associated with feature extraction. In this work, we take a more realistic approach, and argue that fe…
Encrypted Traffic Detection in Resource Constrained IoT Networks: A Diffusion Model and LLM Integrated Framework
Jilin University（吉林大学）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）；Nanyang Technological University（南洋理工大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
TraGe: A Generic Packet Representation for Traffic Classification Based on Header-Payload Differences
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Purple Mountain Laboratories（紫金山实验室）；Nanjing Institute of InforSuperBahn（南京信息高速研究院）
