LogFold:利用结构化令牌和混合编码压缩日志
LogFold: Compressing Logs with Structured Tokens and Hybrid Encoding
AI总结:
本文提出LogFold日志压缩方法,通过结构化令牌分析、模式挖掘、混合编码和打包四个组件,提升压缩效率与效果,实验表明其在压缩比和速度上均优于现有方法。
摘要:
日志对于故障诊断和回顾研究至关重要,导致许多软件组织长期保留日志信息。然而,随着软件系统的增长,生成的日志数据量迅速增加,需要有效的压缩方法。除了通用压缩器(如Gzip、Bzip2)外,许多近期研究开发了日志特定的压缩算法,但因(1)忽视某些复杂令牌内的冗余性,以及(2)缺乏针对不同令牌类型的细粒度编码策略,性能欠佳。本文揭示了结构化令牌中的新冗余模式,并提出了一种新型的类型感知编码策略以提高日志压缩。基于这一见解,我们引入LogFold,一种新的日志压缩方法,包含四个组件:令牌分析器用于将令牌分类为结构化、非结构化或静态类型;处理器基于令牌分隔符骨架挖掘结构化令牌中的重复模式;混合编码器根据令牌类型定制数据表示;打包器将输出压缩成档案文件。对16个公开日志数据集的广泛实验表明,LogFold超越了现有最先进基线,平均压缩比提高11.11%,压缩速度为9.842 MB/s。消融研究进一步表明每个组件的重要性。我们还进行了敏感性分析,验证LogFold在各种内部设置下的鲁棒性和稳定性。
英文摘要:
Logs are essential for diagnosing failures and conducting retrospective studies, leading many software organizations to retain log messages for a long time. Nevertheless, the volume of generated log data grows rapidly as software systems grow, necessitating an effective compression method. Apart from general-purpose compressors (e.g., Gzip, Bzip2), many recent studies developed log-specific compression algorithms, but they offer suboptimal performance because of (1) overlooking redundancies within certain complex tokens, and (2) lacking a fine-grained encoding strategy for diverse token types. This work uncovers a new redundancy pattern in structured tokens and proposes a new type-aware encoding strategy to improve log compression. Building on this insight, we introduce LogFold, a novel log compression method consisting of four components: a token analyzer to classifies tokens as structured, unstructured, or static types; a processor that mines recurring patterns within structured tokens based on their delimiter skeletons; a hybrid encoder that tailors data representation according to token types; and a packer that compresses the output into an archive file. Extensive experiments on 16 public log datasets demonstrate that LogFold surpasses state-of-the-art baselines, achieving average compression ratio improvements by 11.11%, with a compression speed of 9.842 MB/s. Ablation studies further indicate the importance of each component. We also conduct sensitivity analyses to verify LogFold's robustness and stability across various internal settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。