基于Spark的异常检测:端口和网络扫描的实例
Spark-Based Anomaly Detection: the Case of Port and Net Scan
摘要:
最常见的两种网络异常是端口和网络扫描。在这项工作中,我们提出并分析了传统的方法检测网络扫描和端口扫描所获得的结果。我们使用一个简单的基于阈值的算法,在流级别工作,并将其调整为在Apache Spark上执行。使用大数据分析技术可以显著缩短算法的执行时间,即使在当前的高速网络中也可以使用。本文介绍了我们的方法,并提出了一个实验分析的检测性能和执行时间。我们使用真实的流量跟踪MAWI存档和MAWILab异常检测器与我们的结果进行比较。分析表明,i)我们传统的基于阈值的算法已经能够实现比MAWILab更高的检测性能(在95%的最佳阈值情况下),目前被认为是该领域的黄金标准; ii)执行时间比跟踪时间短得多,这使得它也可以在实时的中使用。此外,对于每个流量跟踪,我们为研究社区提供了一个新的标记数据集,通过与MAWILab的比较进行验证,并扩展了其他未检测到的异常。我们每天在我们的项目网站上发布更新的数据集。
英文摘要:
The two most spread network anomalies are port and net scan. In this work, we present and analyze the results obtained by traditional approaches for the detection of net scan and port scans. We use a simple threshold-based algorithm, working at flow-level and adapt it for the execution on Apache Spark. The use of Big Data Analytics technologies allows to significantly the execution times of the algorithm so to be used even in current, high-speed networks. The paper describes our approach and presents an experimental analysis in terms of detection performance and execution time. We use real traffic traces from MAWI archive and MAWILab anomaly detectors to compare with our results. The analysis shows that i) our traditional threshold-based algorithm is already able to achieve detection performance higher than MAWILab (in 95% of the considered cases with the best threshold value), currently considered the gold standard in the field; ii) the execution time is much shorter than the trace time, which makes it usable also in real time. Moreover, for each traffic trace we provide the research community with a new labeled dataset, validated by comparisons with MAWILab and extended with other anomalies not detected by it. We publish an updated dataset every day at our project website.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。