arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1806.04356cs.SE
收藏

在线日志分析的有向无环图方法

A Directed Acyclic Graph Approach to Online Log Parsing

Pinjia He, Jieming Zhu, Pengcheng Xu, Zibin Zheng, Michael R. Lyu

摘要:

在现代软件系统管理中,数据库被广泛使用,因为它们通常是在运行时记录系统事件的唯一可访问的数据。近年来,由于日志规模的不断增加,数据采掘技术经常被用来帮助开发人员和运营商进行系统可靠性管理。一个典型的基于日志的系统可靠性管理过程是首先解析日志消息,因为它们的非结构化格式;并应用数据采掘技术在解析的日志上获得关键的系统行为信息。现有的研究大多集中于离线日志解析,需要以批处理方式解析日志。然而,软件系统,特别是分布式系统,需要在线监控和维护。因此,非常需要能够以流式方式解析日志消息的日志解析器。为了解决这个问题,我们提出了一个在线日志解析方法,即排水,基于有向无环图,编码专门设计的规则进行解析。Drain可以自动为新系统生成有向无环图,并根据传入的日志消息更新该图。此外,Drain允许开发人员在没有预定义参数的情况下使用Drain,从而使他们摆脱了参数调优的负担。为了评估排水的性能,我们收集了11个日志数据集生成的真实世界的系统,从分布式系统,Web应用程序,超级计算机,操作系统,独立的软件。实验结果表明,Drain在所有11个数据集上具有最高的准确率。此外,Drain在运行时间上比最先进的在线解析器提高了37.15\% $\sim$ 97.14\%。我们还进行了一个基于日志的异常检测任务的案例研究,使用排水在解析步骤,这决定了它在系统可靠性管理的有效性。

英文摘要:

Logs are widely used in modern software system management because they are often the only data accessible that record system events at runtime. In recent years, because of the ever-increasing log size, data mining techniques are often utilized to help developers and operators conduct system reliability management. A typical log-based system reliability management procedure is to first parse log messages because of their unstructured format; and apply data mining techniques on the parsed logs to obtain critical system behavior information. Most of existing research studies focus on offline log parsing, which need to parse logs in batch mode. However, software systems, especially distributed systems, require online monitoring and maintenance. Thus, a log parser that can parse log messages in a streaming manner is highly in demand. To address this problem, we propose an online log parsing method, namely Drain, based on directed acyclic graph, which encodes specially designed rules for parsing. Drain can automatically generate a directed acyclic graph for a new system and update the graph according to the incoming log messages. Besides, Drain frees developers from the burden of parameter tuning by allowing them use Drain with no pre-defined parameters. To evaluate the performance of Drain, we collect 11 log datasets generated by real-world systems, ranging from distributed systems, Web applications, supercomputers, operating systems, to standalone software. The experimental results show that Drain has the highest accuracy on all 11 datasets. Moreover, Drain obtains 37.15\%$\sim$ 97.14\% improvement in the running time over the state-of-the-art online parsers. We also conduct a case study on a log-based anomaly detection task using Drain in the parsing step, which determines its effectiveness in system reliability management.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑