Logram：使用n-Gram字典的高效日志解析
Logram: Efficient Log Parsing Using n-Gram Dictionaries
摘要：
软件系统通常在其日志中记录重要的运行时信息。帮助从业者理解系统运行时的行为和诊断现场故障。由于日志通常非常大，因此需要自动化日志分析来帮助从业人员进行软件操作和维护工作。通常，自动日志分析的第一步是日志解析，即，将非结构化原始日志转换为结构化数据。然而，日志解析是具有挑战性的，因为日志是由源代码中的静态模板产生的（即，日志记录语句），但是当解析日志时，模板通常是不可访问的。以前的工作提出了自动日志解析方法，已经达到了很高的准确性。然而，随着日志量在云计算时代的快速增长，效率成为日志解析中的主要关注点。在这项工作中，我们提出了一个自动化的日志解析方法，Logram，它利用n-gram字典来实现高效的日志解析。我们在16个公共日志数据集上评估了Logram，并将Logram与五种最先进的日志解析方法进行了比较。我们发现Logram实现了与现有最佳方法相似的解析准确性，同时在效率上优于这些方法（即，1.8比第二快的方法快5.1倍）。此外，我们在Spark上部署了Logram，我们发现Logram随着Spark节点的数量（例如，具有接近线性的可缩放性）而不牺牲解析准确性。此外，我们证明了Logram可以支持有效的在线日志解析，实现类似的解析结果和效率与离线模式。
英文摘要：
Software systems usually record important runtime information in their logs. Logs help practitioners understand system runtime behaviors and diagnose field failures. As logs are usually very large in size, automated log analysis is needed to assist practitioners in their software operation and maintenance efforts. Typically, the first step of automated log analysis is log parsing, i.e., converting unstructured raw logs into structured data. However, log parsing is challenging, because logs are produced by static templates in the source code (i.e., logging statements) yet the templates are usually inaccessible when parsing logs. Prior work proposed automated log parsing approaches that have achieved high accuracy. However, as the volume of logs grows rapidly in the era of cloud computing, efficiency becomes a major concern in log parsing. In this work, we propose an automated log parsing approach, Logram, which leverages n-gram dictionaries to achieve efficient log parsing. We evaluated Logram on 16 public log datasets and compared Logram with five state-of-the-art log parsing approaches. We found that Logram achieves a similar parsing accuracy to the best existing approaches while outperforms these approaches in efficiency (i.e., 1.8 to 5.1 times faster than the second fastest approaches). Furthermore, we deployed Logram on Spark and we found that Logram scales out efficiently with the number of Spark nodes (e.g., with near-linear scalability) without sacrificing parsing accuracy. In addition, we demonstrated that Logram can support effective online parsing of logs, achieving similar parsing results and efficiency with the offline mode.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
