日志是半结构化文本文件,表示软件运行期间的执行路径和状态。因此,检测软件日志中的异常能够反映软件执行路径或状态中的异常,这已成为软件工程中一个值得关注的问题。与许多先前工作一样,我们使用 LSTM,并在 LSTM 之上提出了一种基于孪生网络的新型异常检测方法。本文还在 Hadoop 分布式文件系统(HDFS)日志数据集上对该方法进行了真实有效的验证。据我们所知,所提出的方法在同一数据集上以 0.996 的 F1 分数优于其他方法,在该数据集上取得了新的最先进性能。除主要方法外,我们还引入了一种新颖的训练对生成算法,该算法将生成的训练对减少了 3000 倍,同时保持了 F1 分数,仅从 0.996 略微下降到 0.995。此外,我们通过将孪生网络与传统前馈神经网络相结合,提出了一种混合模型,使端到端训练成为可能,从而减少了搭建基于深度学习的日志异常检测器所需的工程工作量。进一步地,我们通过在合成演化的日志序列上评估模型,考察了该方法对日志演化的鲁棒性;在噪声比例为 20% 时,我们获得了 0.95 的 F1 分数。最后,我们深入探讨了孪生网络的一些附带优势。据此,我们引入了一种在运行时无需标签要求即可监测日志演化的方法。此外,我们还提出了一种可视化技术,有助于人工管理日志异常检测。
英文摘要:
Logs are semi-structured text files that represent software's execution paths and states during its run-time. Therefore, detecting anomalies in software logs reflect anomalies in the software's execution path or state. So, it has become a notable concern in software engineering. We use LSTM like many prior works, and on top of LSTM, we propose a novel anomaly detection approach based on the Siamese network. This paper also provides an authentic validation of the approach on the Hadoop Distributed File System (HDFS) log dataset. To the best of our knowledge, the proposed approach outperforms other methods on the same dataset at the F1 score of 0.996, resulting in a new state-of-the-art performance on the dataset. Along with the primary method, we introduce a novel training pair generation algorithm that reduces generated training pairs by the factor of 3000 while maintaining the F1 score, merely a modest decay from 0.996 to 0.995. Additionally, we propose a hybrid model by combining the Siamese network with a traditional feedforward neural network to make end-to-end training possible, reducing engineering effort in setting up a deep-learning-based log anomaly detector. Furthermore, we examine our method's robustness to log evolutions by evaluating the model on synthetically evolved log sequences; we got the F1 score of 0.95 at the noise ratio of 20%. Finally, we dive deep into some of the side benefits of the Siamese network. Accordingly, we introduce a method of monitoring the evolutions of logs without label requirements at run-time. Additionally, we present a visualization technique that facilitates human administrations of log anomaly detection.
MINES: Explainable Anomaly Detection through Web API Invariant Inference
National University of Singapore(国立新加坡大学);Shanghai Jiao Tong University(上海交通大学);Singapore Management University(新加坡管理学院);GovTech Singapore(新加坡政府科技局);Chongqing University(重庆大学)
Architecting software monitors for control-flow anomaly detection through large language models and conformance checking
University of Naples Federico II(那不勒斯费德里科二世大学);University of Applied Sciences and Arts of Southern Switzerland(瑞士南部应用科学与艺术大学);University of Florence(佛罗伦萨大学);Linnaeus University(林奈大学)
Anomaly Detection and Root Cause Analysis for Microservice Systems
Viet Nam National University Ho Chi Minh City - University of Technology (HCMUT)(越南国家大学胡志明城-技术大学(HCMUT));School of Computing Technologies(计算技术学院);College of Science, Technology, Engineering and Maths(科学、技术、工程和数学学院);Royal Melbourne Institute of Technology (RMIT University)(皇家墨尔本理工学院(RMIT大学))