arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.06485cs.SEcs.CLcs.LG
收藏

大规模云系统中的知识感知告警聚合:一种混合方法

Knowledge-aware Alert Aggregation in Large-scale Cloud Systems: a Hybrid Approach

The Chinese University of Hong Kong(香港中文大学);Huawei Cloud Computing Technology Co., Ltd(华为云计算技术有限公司)

Jinxi Kuang, Jinyang Liu, Junjie Huang, Renyi Zhong, Jiazhen Gu, Lan Yu, Rui Tan, Zengyin Yang, Michael R. Lyu

AI总结:

本文提出 COLA 混合告警聚合框架,结合关联挖掘与 LLM 推理,并引入 SOP 外部知识,在三个生产数据集上 F1 达 0.901–0.930,已部署于 Cloud X。

摘要:

由于云系统的规模和复杂性,一次系统故障会触发“告警风暴”,即大量相互关联的告警。尽管这些告警可以追溯到少数几个根本原因,但数量之多使得人工处理不可行。因此,告警聚合对于帮助工程师聚焦根本原因并促进故障解决至关重要。现有方法通常利用基于语义相似性的方法或统计方法来聚合告警。然而,基于语义相似性的方法忽视了告警之间的因果原理,而统计方法则难以处理不频繁出现的告警。为解决这些局限,我们引入利用外部知识,即告警的标准操作程序(Standard Operation Procedure,SOP)作为补充。我们提出了 COLA,这是一种基于关联挖掘和 LLM(Large Language Model,大语言模型)推理的新型混合方法,用于在线告警聚合。关联挖掘模块有效捕捉告警之间的时间和空间关系,并以高效方式度量其相关性。随后,只有低置信度的不确定告警对才会被转发给 LLM 推理模块进行详细分析。这种混合设计同时利用了针对频繁告警的统计证据以及计算密集型 LLM 的推理能力,确保 COLA 在实际场景中处理大量告警时具备整体效率。我们在从一个大规模云平台生产环境收集的三个数据集上评估了 COLA。实验结果表明,COLA 取得了 0.901 至 0.930 的 F1 分数,优于最先进的方法,并实现了相当的效率。我们还分享了在真实云系统 Cloud X 中部署 COLA 的经验。

英文摘要:

Due to the scale and complexity of cloud systems, a system failure would trigger an "alert storm", i.e., massive correlated alerts. Although these alerts can be traced back to a few root causes, the overwhelming number makes it infeasible for manual handling. Alert aggregation is thus critical to help engineers concentrate on the root cause and facilitate failure resolution. Existing methods typically utilize semantic similarity-based methods or statistical methods to aggregate alerts. However, semantic similarity-based methods overlook the causal rationale of alerts, while statistical methods can hardly handle infrequent alerts. To tackle these limitations, we introduce leveraging external knowledge, i.e., Standard Operation Procedure (SOP) of alerts as a supplement. We propose COLA, a novel hybrid approach based on correlation mining and LLM (Large Language Model) reasoning for online alert aggregation. The correlation mining module effectively captures the temporal and spatial relations between alerts, measuring their correlations in an efficient manner. Subsequently, only uncertain pairs with low confidence are forwarded to the LLM reasoning module for detailed analysis. This hybrid design harnesses both statistical evidence for frequent alerts and the reasoning capabilities of computationally intensive LLMs, ensuring the overall efficiency of COLA in handling large volumes of alerts in practical scenarios. We evaluate COLA on three datasets collected from the production environment of a large-scale cloud platform. The experimental results show COLA achieves F1-scores from 0.901 to 0.930, outperforming state-of-the-art methods and achieving comparable efficiency. We also share our experience in deploying COLA in our real-world cloud system, Cloud X.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑