OMLog:面向演化系统的基于元学习的在线日志异常检测
OMLog: Online Log Anomaly Detection for Evolving System with Meta-learning
AI总结:
OMLog提出一种半监督在线元学习方法,通过最大均值差异检测分布偏移并自适应切换在线细粒度检测与离线快速推理,在HDFS和BGL数据集上仅用正常日志训练即取得93.7%和64.9%的F1-Score,超越SOTA方法。
摘要:
日志异常检测(LAD)对于确保软件系统的安全稳定运行至关重要。尽管当前的LAD方法在应对不稳定的日志事件和时间序列模式所带来的挑战方面展现出显著潜力,但它们在检测效率和泛化能力上的局限性,在处理演化系统时构成了严峻挑战。为了构建一个实时且可靠的在线日志异常检测模型,我们提出了OMLog,一种半监督在线元学习方法,以有效应对由日志事件类型和频率变化引起的分布偏移问题。具体而言,我们引入了一种基于最大均值差异的分布偏移检测方法,用于识别未见日志序列中的分布变化。根据识别出的分布差距,该方法能够自动触发在线细粒度检测或离线快速推理。此外,我们设计了一种基于元学习的在线学习机制,该机制能够在特征空间中有效学习日志序列的高度重复模式,从而增强模型对演化数据的泛化能力。在两个公开可用的日志数据集HDFS和BGL上进行的大量实验验证了OMLog方法的有效性。当仅使用正常日志序列进行训练时,所提方法分别取得了93.7%和64.9%的F1-Score,超越了当前最先进(SOTA)LAD方法的性能,并展现出优越的检测效率。
英文摘要:
Log anomaly detection (LAD) is essential to ensure safe and stable operation of software systems. Although current LAD methods exhibit significant potential in addressing challenges posed by unstable log events and temporal sequence patterns, their limitations in detection efficiency and generalization ability present a formidable challenge when dealing with evolving systems. To construct a real-time and reliable online log anomaly detection model, we propose OMLog, a semi-supervised online meta-learning method, to effectively tackle the distribution shift issue caused by changes in log event types and frequencies. Specifically, we introduce a maximum mean discrepancy-based distribution shift detection method to identify distribution changes in unseen log sequences. Depending on the identified distribution gap, the method can automatically trigger online fine-grained detection or offline fast inference. Furthermore, we design an online learning mechanism based on meta-learning, which can effectively learn the highly repetitive patterns of log sequences in the feature space, thereby enhancing the generalization ability of the model to evolving data. Extensive experiments conducted on two publicly available log datasets, HDFS and BGL, validate the effectiveness of the OMLog approach. When trained using only normal log sequences, the proposed approach achieves the F1-Score of 93.7\% and 64.9\%, respectively, surpassing the performance of the state-of-the-art (SOTA) LAD methods and demonstrating superior detection efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。