Lemur：基于熵采样与思维链合并的日志解析
Lemur: Log Parsing with Entropy Sampling and Chain-of-Thought Merging
AI总结：
针对现有日志解析依赖人工规则且忽略语义信息的问题，本文提出Lemur框架，采用熵采样聚类日志和思维链合并模板，利用LLM语义理解，在大规模公共数据集上达到最先进性能。
摘要：
广泛软件系统产生的日志对于监控系统行为至关重要。先进的日志分析有助于检测、告警和诊断系统故障。日志解析，即将原始日志消息转换为结构化模板，是日志分析自动化中的关键阶段。现有的日志解析器由于依赖人工制定的规则，无法识别正确的模板。此外，这些方法侧重于统计特征，而忽略了日志消息中的语义信息。为解决这些挑战，我们引入了一种前沿的日志解析框架，名为Lemur，它采用熵采样和思维链合并技术。具体而言，为摒弃繁琐的人工规则，我们提出了一种受信息熵启发的新型采样方法，该方法能高效地聚类典型日志。此外，为增强日志模板的合并，我们设计了一种针对大型语言模型（LLMs）的思维链方法。LLMs展现出卓越的语义理解能力，能巧妙地区分参数和不变量标记。我们在大规模公共数据集上进行了实验。广泛的评估表明，Lemur实现了最先进的性能和令人印象深刻的效率。代码可在https://github.com/zwpride/lemur获取。
英文摘要：
Logs produced by extensive software systems are integral to monitoring system behaviors. Advanced log analysis facilitates the detection, alerting, and diagnosis of system faults. Log parsing, which entails transforming raw log messages into structured templates, constitutes a critical phase in the automation of log analytics. Existing log parsers fail to identify the correct templates due to reliance on human-made rules. Besides, these methods focus on statistical features while ignoring semantic information in log messages. To address these challenges, we introduce a cutting-edge \textbf{L}og parsing framework with \textbf{E}ntropy sampling and chain-of-thought \textbf{M}erging (\model{}). Specifically, to discard the tedious manual rules, we propose a novel sampling method inspired by information entropy, which efficiently clusters typical logs. Furthermore, to enhance the merging of log templates, we design a chain-of-thought method for large language models (LLMs). LLMs exhibit exceptional semantic comprehension and deftly distinguish between parameters and invariant tokens. We have conducted experiments on large-scale public datasets. Extensive evaluation demonstrates that \model{} achieves state-of-the-art performance and impressive efficiency. The Code is available at https://github.com/zwpride/lemur.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
