arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.07324cs.SEcs.LG
收藏

OneLog:迈向软件日志异常检测中的端到端训练

OneLog: Towards End-to-End Training in Software Log Anomaly Detection

University of Oulu(奥卢大学);University of Helsinki(赫尔辛基大学)

Shayan Hashemi, Mika Mäntylä

AI总结:

本文提出 OneLog,用单个字符级 CNN 端到端处理日志文本、数字与标点,取代传统四阶段流程,并在六个数据集及多种项目设置下验证其先进性能与跨数据集泛化能力。

摘要:

随着在线服务、物联网设备和面向 DevOps 的软件开发的增长,软件日志异常检测正变得越来越重要。以往工作主要遵循传统的四阶段架构(预处理器、解析器、向量化器和分类器)。本文提出 OneLog,它使用单个深度神经网络(DNN),而非多个独立组件。OneLog 在字符层面利用卷积神经网络(CNN),将以往工作中被移除的数字、数值和标点符号与主要自然语言文本一并纳入考虑。我们在六个基于消息和基于序列的数据集上评估了该方法:HDFS、Hadoop、BGL、Thunderbird、Spirit 和 Liberty。我们在单项目、多项目和跨项目设置下对 OneLog 进行实验。OneLog 在我们的数据集上提供了最先进的性能。OneLog 能够在训练期间同时利用多个项目数据集,这表明我们的模型能够在数据集之间泛化。多项目训练还提升了 OneLog 的性能,使其在单个项目可用训练数据有限时非常理想。我们还发现,仅使用一个项目对(Liberty 和 Spirit)即可进行跨项目异常检测。对模型内部机制的分析表明,一条日志具有多种检测异常的模式,并且模型会学习经过人工验证的日志消息解析规则。我们得出结论:基于字符的 CNN 是日志异常检测中端到端学习的一种有前景的方法。它们在多个数据集上提供良好的性能和泛化能力。论文录用后,我们将公开脚本。

英文摘要:

With the growth of online services, IoT devices, and DevOps-oriented software development, software log anomaly detection is becoming increasingly important. Prior works mainly follow a traditional four-staged architecture (Preprocessor, Parser, Vectorizer, and Classifier). This paper proposes OneLog, which utilizes a single Deep Neural Network (DNN) instead of multiple separate components. OneLog harnesses Convolutional Neural Networks (CNN) at the character level to take digits, numbers, and punctuations, which were removed in prior works, into account alongside the main natural language text. We evaluate our approach in six message- and sequence-based data sets: HDFS, Hadoop, BGL, Thunderbird, Spirit, and Liberty. We experiment with Onelog with single-, multi-, and cross-project setups. Onelog offers state-of-the-art performance in our datasets. Onelog can utilize multi-project datasets simultaneously during training, which suggests our model can generalize between datasets. Multi-project training also improves Onelog performance making it ideal when limited training data is available for an individual project. We also found that cross-project anomaly detection is possible with a single project pair (Liberty and Spirit). Analysis of model internals shows that one log has multiple modes of detecting anomalies and that the model learns manually validated parsing rules for the log messages. We conclude that character-based CNNs are a promising approach toward end-to-end learning in log anomaly detection. They offer good performance and generalization over multiple datasets. We will make our scripts publicly available upon the acceptance of this paper.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑