FlexParser --在不断变化的世界中提供连续结果的自适应日志文件解析器
FlexParser -- the adaptive log file parser for continuous results in a changing world
摘要:
任何现代系统都将事件写入文件,称为日志文件。这些报告载有关键信息,需要进行各种分析。示例范围从网络安全、使用分析上的入侵检测到故障排除。在数据分析成为可能之前,需要首先从半结构化日志消息中提取所需的信息。最先进的事件解析通常假设静态日志事件。然而,任何现代系统都是一致更新的,并且随着更新,日志文件结构也会改变。我们称这些变化为“突变”,并研究不同的突变情况下的分析性能。最新的研究发现突变使用异常检测事后剖析,但是,并没有涵盖实际的连续解析。因此,我们提出了一种新的和灵活的解析器,称为FlexParser,它可以提取所需的值,尽管日志消息的逐渐变化。它意味着基本的文本预处理,然后是有监督的深度学习方法。我们训练一个有状态的LSTM来解析每个数据集的一个事件。有状态性强制模型在多个示例中学习日志消息结构。我们的模型在七个不同的、公开可用的日志文件数据集和各种突变上进行了测试。它的平均F1分数为0.98,优于其他深度学习方法以及最先进的无监督解析器。
英文摘要:
Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi-structured log messages. State-of-the-art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1-Score of 0.98, it outperforms other Deep Learning methods as well as state-of-the-art unsupervised parsers.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。