arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.05760cs.SE
收藏

解锁数字的力量:通过数值令牌解析实现日志压缩

Unlocking the Power of Numbers: Log Compression via Numeric Token Parsing

Siyu Yu, Yifan Wu, Ying Li, Pinjia He

AI总结:

针对基于解析器的日志压缩器存在的解析与压缩目标不一致、性能不稳定和处理时间长等问题,本文提出Denum,通过数值令牌解析模块和字符串处理模块分别处理数值令牌和剩余内容,在16个日志数据集上实现了显著更高的压缩比和更快的压缩速度。

摘要:

近年来,基于解析器的日志压缩器已被广泛探索,因为日志量的爆炸性增长使得通用压缩器的压缩性能不尽如人意。这些基于解析器的压缩器通过根据解析结果对日志进行分组来预处理日志,然后将预处理后的文件输入通用压缩器。然而,基于解析器的压缩器存在其局限性。首先,解析和压缩的目标不一致,因此日志的固有特性未被充分利用。此外,基于解析器的压缩器的性能依赖于样本日志,因此非常不稳定。而且,基于解析器的压缩器通常需要较长的处理时间。为了解决这些局限性,我们提出了Denum,一种简单、通用的日志压缩器,具有高压缩比和高速度。核心见解是日志中的大多数令牌是数值令牌(即纯数字、仅包含数字和特殊字符的令牌以及数值变量),对它们进行有效压缩对于日志压缩至关重要。具体而言,Denum包含一个数值令牌解析模块,该模块提取所有数值令牌并应用定制处理方法(例如存储递增数字如时间戳的差值),以及一个字符串处理模块,该模块处理不含数字的剩余日志内容。两个模块处理后的文件随后作为输入送入通用压缩器,由其输出最终压缩结果。Denum已在16个日志数据集上进行了评估,与基线相比,它实现了8.7%-434.7%更高的平均压缩比和2.6倍-37.7倍更快的平均压缩速度(即26.2MB/S)。此外,将Denum的数值令牌解析集成到现有日志压缩器中,可以使其平均压缩比提高11.8%,并实现37%更快的平均压缩速度。

英文摘要:

Parser-based log compressors have been widely explored in recent years because the explosive growth of log volumes makes the compression performance of general-purpose compressors unsatisfactory. These parser-based compressors preprocess logs by grouping the logs based on the parsing result and then feed the preprocessed files into a general-purpose compressor. However, parser-based compressors have their limitations. First, the goals of parsing and compression are misaligned, so the inherent characteristics of logs were not fully utilized. In addition, the performance of parser-based compressors depends on the sample logs and thus it is very unstable. Moreover, parser-based compressors often incur a long processing time. To address these limitations, we propose Denum, a simple, general log compressor with high compression ratio and speed. The core insight is that a majority of the tokens in logs are numeric tokens (i.e. pure numbers, tokens with only numbers and special characters, and numeric variables) and effective compression of them is critical for log compression. Specifically, Denum contains a Numeric Token Parsing module, which extracts all numeric tokens and applies tailored processing methods (e.g. store the differences of incremental numbers like timestamps), and a String Processing module, which processes the remaining log content without numbers. The processed files of the two modules are then fed as input to a general-purpose compressor and it outputs the final compression results. Denum has been evaluated on 16 log datasets and it achieves an 8.7%-434.7% higher average compression ratio and 2.6x-37.7x faster average compression speed (i.e. 26.2MB/S) compared to the baselines. Moreover, integrating Denum's Numeric Token Parsing into existing log compressors can provide an 11.8% improvement in their average compression ratio and achieve 37% faster average compression speed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑