arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.01896cs.CLcs.IR
收藏

LogEval:面向大型语言模型日志分析的综合基准套件

LogEval: A Comprehensive Benchmark Suite for Large Language Models In Log Analysis

Nankai University(南开大学);Tsinghua University(清华大学);Huawei(华为)

Tianyu Cui, Shiyu Ma, Ziang Chen, Tong Xiao, Shimin Tao, Yilun Liu, Shenglin Zhang, Duoming Lin, Changchang Liu, Yuzhe Cai, Weibin Meng, Yongqian Sun, Dan Pei

AI总结:

本文提出LogEval综合基准,从日志解析、异常检测、故障诊断和摘要等任务系统评估主流LLM,并分析量化、提示策略及中英双语表现。

摘要:

日志分析对于确保信息系统有序、稳定运行至关重要,尤其是在人工智能用于IT运维(AIOps)领域。大型语言模型(LLM)已在自然语言处理任务中展现出显著潜力。在AIOps领域,它们擅长异常检测、故障根因分析、运维脚本生成和告警信息摘要等任务。然而,当前LLM在日志分析任务上的性能仍未得到充分验证。为弥补这一空白,我们推出LogEval,这是一个首次旨在评估LLM在多种日志分析任务中能力的综合基准套件。该基准涵盖日志解析、日志异常检测、日志故障诊断和日志摘要等任务。LogEval对每项任务使用4,000条公开可用的日志数据条目进行评估,并为每项任务采用15种不同提示,以确保评估全面且公平。通过严格评估主流LLM,我们展示了各类LLM技术对日志分析性能的影响,重点关注自洽性和少样本上下文学习等方面。我们还讨论了与模型量化、中英文问答评估以及提示工程相关的发现。这些发现揭示了LLM在多语言环境中的优势与弱点,以及不同提示策略的有效性。研究针对不同任务采用多种评估方法,以准确衡量LLM在日志分析中的性能,确保评估的全面性。从LogEval评估中获得的见解揭示了LLM在日志分析任务中的优势与局限,为研究人员和从业者提供了宝贵指导。

英文摘要:

Log analysis is crucial for ensuring the orderly and stable operation of information systems, particularly in the field of Artificial Intelligence for IT Operations (AIOps). Large Language Models (LLMs) have demonstrated significant potential in natural language processing tasks. In the AIOps domain, they excel in tasks such as anomaly detection, root cause analysis of faults, operations and maintenance script generation, and alert information summarization. However, the performance of current LLMs in log analysis tasks remains inadequately validated. To address this gap, we introduce LogEval, a comprehensive benchmark suite designed to evaluate the capabilities of LLMs in various log analysis tasks for the first time. This benchmark covers tasks such as log parsing, log anomaly detection, log fault diagnosis, and log summarization. LogEval evaluates each task using 4,000 publicly available log data entries and employs 15 different prompts for each task to ensure a thorough and fair assessment. By rigorously evaluating leading LLMs, we demonstrate the impact of various LLM technologies on log analysis performance, focusing on aspects such as self-consistency and few-shot contextual learning. We also discuss findings related to model quantification, Chinese-English question-answering evaluation, and prompt engineering. These findings provide insights into the strengths and weaknesses of LLMs in multilingual environments and the effectiveness of different prompt strategies. Various evaluation methods are employed for different tasks to accurately measure the performance of LLMs in log analysis, ensuring a comprehensive assessment. The insights gained from LogEvals evaluation reveal the strengths and limitations of LLMs in log analysis tasks, providing valuable guidance for researchers and practitioners.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑