arXivDaily arXiv每日学术速递 周一至周五更新
看起来今天可以划水了
arXiv 2409.02474cs.SEcs.CL
收藏

面向日志解析的大语言模型比较研究

A Comparative Study on Large Language Models for Log Parsing

Simula Research Laboratory(西玛拉研究实验室);BI Norwegian Business School(BI挪威商学院)

Merve Astekin, Max Hort, Leon Moonen

AI总结:

本文比较了六种最新大语言模型在日志解析任务上的表现,发现免费开放的代码专用模型(如CodeLlama)可超越付费模型GPT-3.5,正确提取模板数高出10%,表明小型免费LLMs能有效辅助日志解析。

摘要:

背景:日志消息提供了关于软件系统状态的有价值信息。这些信息以非结构化方式呈现,需要应用自动化方法来提取相关参数。为了简化这一过程,可以应用日志解析,将日志消息转换为结构化的日志模板。语言模型的最新进展引发了多项研究,将ChatGPT应用于日志解析任务并取得了有前景的结果。然而,其他最先进的大语言模型(LLMs)在日志解析任务上的表现仍不清楚。目的:在本研究中,我们调查了当前最先进的LLMs执行日志解析的能力。方法:我们选取了六种近期LLMs,包括付费专有模型(GPT-3.5、Claude 2.1)和四种免费使用的开放模型,并比较它们在从多个成熟开源项目中获取的系统日志上的表现。我们设计了两种不同的提示方法,并将LLMs应用于16个不同项目中的1,354个日志模板。我们评估了它们的有效性,包括正确识别模板的数量,以及生成的模板与真实模板之间的句法相似度。结果:我们发现免费使用的模型能够与付费模型竞争,其中CodeLlama比GPT-3.5多正确提取了10%的日志模板。此外,我们提供了关于语言模型可用性的定性见解(例如,使用其响应的难易程度)。结论:我们的结果表明,与付费专有竞争对手相比,一些较小的、免费使用的LLMs可以显著辅助日志解析,尤其是代码专用模型。

英文摘要:

Background: Log messages provide valuable information about the status of software systems. This information is provided in an unstructured fashion and automated approaches are applied to extract relevant parameters. To ease this process, log parsing can be applied, which transforms log messages into structured log templates. Recent advances in language models have led to several studies that apply ChatGPT to the task of log parsing with promising results. However, the performance of other state-of-the-art large language models (LLMs) on the log parsing task remains unclear. Aims: In this study, we investigate the current capability of state-of-the-art LLMs to perform log parsing. Method: We select six recent LLMs, including both paid proprietary (GPT-3.5, Claude 2.1) and four free-to-use open models, and compare their performance on system logs obtained from a selection of mature open-source projects. We design two different prompting approaches and apply the LLMs on 1, 354 log templates across 16 different projects. We evaluate their effectiveness, in the number of correctly identified templates, and the syntactic similarity between the generated templates and the ground truth. Results: We found that free-to-use models are able to compete with paid models, with CodeLlama extracting 10% more log templates correctly than GPT-3.5. Moreover, we provide qualitative insights into the usability of language models (e.g., how easy it is to use their responses). Conclusions: Our results reveal that some of the smaller, free-to-use LLMs can considerably assist log parsing compared to their paid proprietary competitors, especially code-specialized models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑