基于深度学习的不稳定日志数据故障识别
Failure Identification from Unstable Log Data using Deep Learning
摘要:
云平台的可靠性具有重要意义,因为社会越来越依赖于在云上运行的复杂软件系统。为了改进它,云提供商正在自动化各种维护任务,并经常考虑故障识别。自动化的前提是可观测性工具的可用性,通常使用系统日志。本文的重点是基于日志的故障识别。这个问题是具有挑战性的,因为日志数据的不稳定性和代码中显式日志失败覆盖的不完整性。为了解决这两个挑战,我们提出了CLog作为故障识别的方法。本文提出的关键思想是基于我们的观察,即通过将日志数据表示为子过程的序列而不是日志事件的序列,减少了不稳定日志数据的影响。CLog引入了一种新的子过程提取方法,该方法使用上下文感知神经网络和聚类方法来提取有意义的子过程。日志事件上下文的直接建模允许相对于突然的上下文变化识别故障,解决了日志故障覆盖不足的挑战。我们的实验结果表明,学习的子过程表示减少了输入的不稳定性,使CLog在故障识别子问题上优于基线- 1)故障检测在F1得分上提高了9-24%,2)故障类型识别在宏观平均F1得分上提高了7%。进一步的分析表明,输入事件序列的不稳定性与检测性能之间存在与模型无关的负相关性。
英文摘要:
The reliability of cloud platforms is of significant relevance because society increasingly relies on complex software systems running on the cloud. To improve it, cloud providers are automating various maintenance tasks, with failure identification frequently being considered. The precondition for automation is the availability of observability tools, with system logs commonly being used. The focus of this paper is log-based failure identification. This problem is challenging because of the instability of the log data and the incompleteness of the explicit logging failure coverage within the code. To address the two challenges, we present CLog as a method for failure identification. The key idea presented herein based is on our observation that by representing the log data as sequences of subprocesses instead of sequences of log events, the effect of the unstable log data is reduced. CLog introduces a novel subprocess extraction method that uses context-aware neural network and clustering methods to extract meaningful subprocesses. The direct modeling of log event contexts allows the identification of failures with respect to the abrupt context changes, addressing the challenge of insufficient logging failure coverage. Our experimental results demonstrate that the learned subprocesses representations reduce the instability in the input, allowing CLog to outperform the baselines on the failure identification subproblems - 1) failure detection by 9-24% on F1 score and 2) failure type identification by 7% on the macro averaged F1 score. Further analysis shows the existent negative correlation between the instability in the input event sequences and the detection performance in a model-agnostic manner.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。