arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.08951cs.CR
收藏

无监督日志异常检测:基于少量独特标记

Unsupervised Log Anomaly Detection with Few Unique Tokens

Lawrence Berkeley National Lab(劳伦斯伯克利国家实验室);Deutsches Elektronen-Synchrotron (DESY)(德国电子同步加速器中心);Technische Universitaet Hamburg (TUHH)(汉堡工业大学)

Antonin Sulc, Annika Eichler, Tim Wilksen

AI总结:

针对欧洲XFEL加速器节点日志,提出基于词嵌入与隐马尔可夫模型的异常检测方法,通过概率比率评分识别偏离常规的日志事件,实现主动预警。

摘要:

本文介绍了一种新颖的方法,用于检测欧洲XFEL加速器控制系统节点日志数据中的异常。有效的异常检测对于为操作员提供每个节点的可用性、状态和潜在问题的清晰理解至关重要,从而确保加速器的平稳运行。由于这些日志的序列性以及缺乏丰富的、节点特定的文本语料库,传统和基于学习的异常检测方法面临显著限制。为解决这一问题,我们提出了一种利用词嵌入表示日志条目,并使用隐马尔可夫模型(HMM)为各个节点建模这些嵌入的典型序列模式的方法。异常通过基于概率比率为单个日志条目评分来识别:该比率比较了包含新条目的日志序列的可能性与其不包含该条目的可能性,从而有效衡量新条目与既定模式的契合程度。高分表明潜在异常偏离了节点的常规行为。该方法作为一个预警系统,提醒操作员注意可能预示潜在问题的异常日志事件,从而促进主动干预。

英文摘要:

This article introduces a novel method for detecting anomalies within log data from control system nodes at the European XFEL accelerator. Effective anomaly detection is crucial for providing operators with a clear understanding of each node's availability, status, and potential problems, thereby ensuring smooth accelerator operation. Traditional and learning-based anomaly detection methods face significant limitations due to the sequential nature of these logs and the lack of a rich, node-specific text corpus. To address this, we propose an approach utilizing word embeddings to represent log entries and a Hidden Markov Model (HMM) to model the typical sequential patterns of these embeddings for individual nodes. Anomalies are identified by scoring individual log entries based on a probability ratio: this ratio compares the likelihood of the log sequence including the new entry against its likelihood without it, effectively measuring how well the new entry fits the established pattern. High scores indicate potential anomalies that deviate from the node's routine behavior. This method functions as a warning system, alerting operators to irregular log events that may signify underlying issues, thereby facilitating proactive intervention.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑