arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.15445cs.SE
收藏

使用元学习的跨系统软件日志异常检测

Cross-System Software Log-based Anomaly Detection Using Meta-Learning

Yuqing Wang, Mika V. Mäntylä, Jesse Nyyssölä, Ke Ping, Liqiang Wang

AI总结:

针对日志异常检测中的标注成本高、日志演变和跨系统适应挑战,提出基于元学习的CroSysLog工具,利用源系统训练并少量样本适应目标系统,在四个超算系统上验证了其高效性与适应性。

摘要:

现代软件系统产生大量日志,是异常检测的重要资源。人工智能运维(AIOps)工具已被开发用于自动化软件系统的日志异常检测过程。该领域公认存在三个实际挑战:高数据标注成本、动态系统中日志的不断演变,以及跨不同系统的适应性。本文提出CroSysLog,一种专门针对这些挑战设计的用于日志事件级异常检测的AIOps工具。遵循先前方法,CroSysLog使用神经表示方法获得对日志的细致理解,并据此为单个日志事件生成表示。CroSysLog可在具有来自开放数据集的充足标注日志的源系统上训练以实现鲁棒性,然后高效适应仅需少量标注日志事件的目标系统以进行有效异常检测。我们使用四个大规模分布式超级计算系统的开放数据集评估CroSysLog:BGL、Thunderbird、Liberty和Spirit。我们使用随机日志分割,保持连续日志事件的时间顺序,从这些系统中训练和评估CroSysLog。这些分割广泛分布在每个系统日志收集持续时间的一至两年跨度内,从而捕捉每个系统中日志的演变特性。结果表明,在Liberty和BGL作为源系统上训练CroSysLog后,CroSysLog能利用每个目标系统的少量标注日志事件高效适应目标系统Thunderbird和Spirit,有效执行这些目标系统的异常检测。结果证明CroSysLog是软件系统运维和维护场景中日志事件级异常检测的一种实用、可扩展且适应性强的工具。

英文摘要:

Modern software systems produce vast amounts of logs, serving as an essential resource for anomaly detection. Artificial Intelligence for IT Operations (AIOps) tools have been developed to automate the process of log-based anomaly detection for software systems. Three practical challenges are widely recognized in this field: high data labeling costs, evolving logs in dynamic systems, and adaptability across different systems. In this paper, we propose CroSysLog, an AIOps tool for log-event level anomaly detection, specifically designed in response to these challenges. Following prior approaches, CroSysLog uses a neural representation approach to gain a nuanced understanding of logs and generate representations for individual log events accordingly. CroSysLog can be trained on source systems with sufficient labeled logs from open datasets to achieve robustness, and then efficiently adapt to target systems with a few labeled log events for effective anomaly detection. We evaluate CroSysLog using open datasets of four large-scale distributed supercomputing systems: BGL, Thunderbird, Liberty, and Spirit. We used random log splits, maintaining the chronological order of consecutive log events, from these systems to train and evaluate CroSysLog. These splits were widely distributed across a one/two-year span of each system's log collection duration, thereby capturing the evolving nature of the logs in each system. Our results show that, after training CroSysLog on Liberty and BGL as source systems, CroSysLog can efficiently adapt to target systems Thunderbird and Spirit using a few labeled log events from each target system, effectively performing anomaly detection for these target systems. The results demonstrate that CroSysLog is a practical, scalable, and adaptable tool for log-event level anomaly detection in operational and maintenance contexts of software systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑