基于隐马尔可夫模型的灵活日志文件解析
Flexible Log File Parsing using Hidden Markov Models
摘要：
我们的目标是模型未知的文件处理。由于日志文件的内容经常随着时间的推移而演变，我们建立了一个动态的统计模型，学习和适应处理和解析规则。首先，我们通过只关注那些导致类似于Vaarandi [10]的期望输出表的频繁模式来限制非结构化文本的数量。其次，我们把发现的频繁模式和输出说明解析表到一个隐马尔可夫模型（HMM）。我们使用这个HMM作为一个特定的，但是，灵活的表示模式的日志文件处理。随着原始日志文件中的变化扭曲了学习的模式，我们的目标是模型自动适应，以保持高质量的输出。在一个系统类型上训练我们的模型后，将模型和产生的解析规则应用于具有稍微不同的日志文件模式的不同系统，我们实现了超过99%的准确率。
英文摘要：
We aim to model unknown file processing. As the content of log files often evolves over time, we established a dynamic statistical model which learns and adapts processing and parsing rules. First, we limit the amount of unstructured text by focusing only on those frequent patterns which lead to the desired output table similar to Vaarandi [10]. Second, we transform the found frequent patterns and the output stating the parsed table into a Hidden Markov Model (HMM). We use this HMM as a specific, however, flexible representation of a pattern for log file processing. With changes in the raw log file distorting learned patterns, we aim the model to adapt automatically in order to maintain high quality output. After training our model on one system type, applying the model and the resulting parsing rule to a different system with slightly different log file patterns, we achieve an accuracy over 99%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
