arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.00409cs.DBcs.SE
收藏

Logzip:通过迭代聚类提取隐藏结构进行日志压缩

Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, Michael R. Lyu

摘要:

系统日志记录了软件系统运行时的详细信息,并被用作围绕软件工程的许多任务的主要数据源。随着现代软件系统向大规模和复杂结构发展,日志已成为工业中快速增长的大数据类型之一。特别地,这种日志在实践中经常需要存储很长时间(例如,一年),以便分析经常出现的问题或跟踪安全问题。然而,归档日志会消耗大量的存储空间和计算资源,这反过来又会带来很高的运营成本。数据压缩对于降低日志存储成本至关重要。传统的压缩工具(例如,gzip)对于一般文本工作得很好,但是对于系统日志不太好。在本文中,我们提出了一种新颖的和有效的日志压缩方法,即logzip。Logzip能够通过快速迭代聚类从原始日志中提取隐藏结构,并进一步生成一致的中间表示,从而实现更有效的压缩。我们在五个不同系统类型的大型日志数据集上评估logzip,总大小为63.6 GB。结果表明,logzip可以节省约一半的存储空间,平均比传统的压缩工具。同时,logzip的设计是高度并行的,只会产生微不足道的开销。此外,我们还分享了我们将logzip应用于华为实际产品的行业经验。

英文摘要:

System logs record detailed runtime information of software systems and are used as the main data source for many tasks around software engineering. As modern software systems are evolving into large scale and complex structures, logs have become one type of fast-growing big data in industry. In particular, such logs often need to be stored for a long time in practice (e.g., a year), in order to analyze recurrent problems or track security issues. However, archiving logs consumes a large amount of storage space and computing resources, which in turn incurs high operational cost. Data compression is essential to reduce the cost of log storage. Traditional compression tools (e.g., gzip) work well for general texts, but are not tailed for system logs. In this paper, we propose a novel and effective log compression method, namely logzip. Logzip is capable of extracting hidden structures from raw logs via fast iterative clustering and further generating coherent intermediate representations that allow for more effective compression. We evaluate logzip on five large log datasets of different system types, with a total of 63.6 GB in size. The results show that logzip can save about half of the storage space on average over traditional compression tools. Meanwhile, the design of logzip is highly parallel and only incurs negligible overhead. In addition, we share our industrial experience of applying logzip to Huawei's real products.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑