Token Interdependency Parsing (Tipping)——快速且准确的日志解析方法
Token Interdependency Parsing (Tipping) -- Fast and Accurate Log Parsing
AI总结：
针对海量日志自动化解析需求，本文提出快速高精度日志解析器Tipping，融合分词、依赖图等技术并支持并行，在准确率、性能上优于现有最优方法，可高效处理大规模日志。
摘要：
过去十年间，软件适配量的显著增长带来了日志数据量的激增，这使得人工日志分析不再具备可行性，也确立了自动化日志分析方法的必要性。而绝大多数自动化分析工具都包含一个用于从日志参数中分离日志模板的组件，这类组件通常被称为“日志解析器”。本文旨在提出一种名为“Tipping”的新型快速高精度日志解析器。Tipping融合了基于规则的分词器、词元依赖图、强连通分量及多种技术，以实现快速、可扩展且精准的日志解析。此外，Tipping支持并行化，可在多个处理核心上运行，效率接近线性水平。我们从准确率、性能以及下游异常检测任务三个维度，将Tipping与其他当前最优的日志解析器进行了对比评估。评估结果显示，Tipping在准确率和性能上均优于现有方法。具体而言，Tipping可在笔记本电脑上于20秒内完成1100万行日志的解析。此外，我们重新实现了过往IpLom算法的并行版本以验证并行处理的效果，该版本成为了速度第二快的解析器。随着日志的体量和复杂度持续增长，软件工程领域需要确保自动化日志分析工具能够跟上需求，具备高效、高精度处理海量日志的能力。Tipping的鲁棒性、通用性、效率和可扩展性使其成为适配现代自动化日志分析任务的可行工具。
英文摘要：
In the last decade, an impressive increase in software adaptions has led to a surge in log data production, making manual log analysis impractical and establishing the necessity for automated methods. Conversely, most automated analysis tools include a component designed to separate log templates from their parameters, commonly referred to as a "log parser". This paper aims to introduce a new fast and accurate log parser, named "Tipping". Tipping combines rule-based tokenizers, interdependency token graphs, strongly connected components, and various techniques to ensure rapid, scalable, and precise log parsing. Furthermore, Tipping is parallelized and capable of running on multiple processing cores with close to linear efficiency. We evaluated Tipping against other state-of-the-art log parsers in terms of accuracy, performance, and the downstream task of anomaly detection. Accordingly, we found that Tipping outperformed existing methods in accuracy and performance in our evaluations. More in-depth, Tipping can parse 11 million lines of logs in less than 20 seconds on a laptop machine. Furthermore, we re-implemented a parallelized version of the past IpLom algorithm to demonstrate the effect of parallel processing, and it became the second-fastest parser. As logs keep growing in volume and complexity, the software engineering community needs to ensure automated log analysis tools keep up with the demand, being capable of efficiently handling massive volumes of logs with high accuracy. Tipping's robustness, versatility, efficiency, and scalability make it a viable tool for the modern automated log analysis task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
