arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17583cs.SEcs.CLcs.LG
收藏

FaultProfIT:大规模云系统中事件工单的层级化故障画像

FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems

The Chinese University of Hong Kong(香港中文大学);Sun Yat-sen University(中山大学);Huawei Cloud Computing Technology Co., Ltd(华为云计算技术有限公司)

Junjie Huang, Jinyang Liu, Zhuangbin Chen, Zhihan Jiang, Yichen LI, Jiazhen Gu, Cong Feng, Zengyin Yang, Yongqiang Yang, Michael R. Lyu

AI总结:

本文提出 FaultProfIT,利用层级引导对比学习自动对云系统事件工单进行故障模式画像,在 CloudA 生产环境中优于现有方法并已实际部署分析超一万个事件。

摘要:

事后分析在云系统的事件管理中至关重要,它为提升系统可靠性与鲁棒性提供了有价值的洞察。在 CloudA,故障模式画像在事后阶段进行,涉及将事件的故障分类为独特的类别,称为故障模式。通过聚合和分析这些故障模式,工程师能够识别常见故障、脆弱组件以及新兴的故障趋势。然而,该过程目前依赖人工标注,存在固有缺陷。一方面,事件数量庞大意味着只有最严重的事件才会被分析,导致故障模式概览出现偏差。另一方面,该任务的复杂性要求广泛的领域知识,从而导致错误和不一致。为了解决这些局限性,我们提出了一种自动化方法,名为 FaultProfIT,用于事件工单的故障模式画像。它利用层级引导的对比学习来训练一个具备层级感知的事件编码器,并利用增强的事件表示来预测故障模式。我们使用 CloudA 的生产事件对 FaultProfIT 进行了评估。结果表明,FaultProfIT 优于最先进的方法。我们的消融研究和分析也验证了层级引导对比学习的有效性。此外,我们已在 CloudA 部署 FaultProfIT 六个月。迄今为止,FaultProfIT 已分析了来自 30 多个云服务的 10,000 多个事件,成功揭示了若干故障趋势,为系统改进提供了依据。

英文摘要:

Postmortem analysis is essential in the management of incidents within cloud systems, which provides valuable insights to improve system's reliability and robustness. At CloudA, fault pattern profiling is performed during the postmortem phase, which involves the classification of incidents' faults into unique categories, referred to as fault pattern. By aggregating and analyzing these fault patterns, engineers can discern common faults, vulnerable components and emerging fault trends. However, this process is currently conducted by manual labeling, which has inherent drawbacks. On the one hand, the sheer volume of incidents means only the most severe ones are analyzed, causing a skewed overview of fault patterns. On the other hand, the complexity of the task demands extensive domain knowledge, which leads to errors and inconsistencies. To address these limitations, we propose an automated approach, named FaultProfIT, for Fault pattern Profiling of Incident Tickets. It leverages hierarchy-guided contrastive learning to train a hierarchy-aware incident encoder and predicts fault patterns with enhanced incident representations. We evaluate FaultProfIT using the production incidents from CloudA. The results demonstrate that FaultProfIT outperforms state-of-the-art methods. Our ablation study and analysis also verify the effectiveness of hierarchy-guided contrastive learning. Additionally, we have deployed FaultProfIT at CloudA for six months. To date, FaultProfIT has analyzed 10,000+ incidents from 30+ cloud services, successfully revealing several fault trends that have informed system improvements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑