PACE-LM：在云事件根本原因分析中利用 GPT-4 通过提示与增强实现校准的置信度估计
PACE-LM: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis
AI总结：
本文提出 PACE-LM，通过提示检索增强的 GPT-4 并结合事件判断与历史参考评分，为黑盒云事件根因预测生成校准置信度。
摘要：
主要云服务提供商已经采用诸如大语言模型等先进的基于 AI 的解决方案，以帮助人工识别云事件的根本原因。尽管 AI 驱动的助手在根本原因分析流程中日益普及，但由于任务本身的内在难度、基于 LLM 的方法容易产生幻觉，以及难以区分这些伪装良好的幻觉，它们在协助值班工程师方面的有效性受到低准确率的限制。为应对这一挑战，我们建议对预测结果执行置信度估计，以帮助值班工程师决定是否采纳模型预测。考虑到许多基于 LLM 的根本原因预测器具有黑盒性质，基于微调或温度缩放的方法并不适用。因此，我们设计了一种创新的置信度估计框架，该框架基于对检索增强型大语言模型（LLM）进行提示，并且只需要根本原因预测器提供极少量信息。该方法包含两个评分阶段：基于 LLM 的置信度估计器首先评估其在面对当前事件时作出判断的置信度，这反映其在参考数据中的“有据可依”程度；随后根据历史参考对根本原因预测进行评分。一个优化步骤将这两个分数结合起来，以给出最终置信度。我们表明，该方法能够为预测的根本原因生成经过校准的置信度估计，并验证了所检索历史数据和提示策略的有用性，以及在不同根本原因预测模型之间的泛化能力。本研究朝着将 LLM 可靠且有效地嵌入云事件管理系统迈出了重要一步。
英文摘要：
Major cloud providers have employed advanced AI-based solutions like large language models to aid humans in identifying the root causes of cloud incidents. Despite the growing prevalence of AI-driven assistants in the root cause analysis process, their effectiveness in assisting on-call engineers is constrained by low accuracy due to the intrinsic difficulty of the task, a propensity for LLM-based approaches to hallucinate, and difficulties in distinguishing these well-disguised hallucinations. To address this challenge, we propose to perform confidence estimation for the predictions to help on-call engineers make decisions on whether to adopt the model prediction. Considering the black-box nature of many LLM-based root cause predictors, fine-tuning or temperature-scaling-based approaches are inapplicable. We therefore design an innovative confidence estimation framework based on prompting retrieval-augmented large language models (LLMs) that demand a minimal amount of information from the root cause predictor. This approach consists of two scoring phases: the LLM-based confidence estimator first evaluates its confidence in making judgments in the face of the current incident that reflects its ``grounded-ness" level in reference data, then rates the root cause prediction based on historical references. An optimization step combines these two scores for a final confidence assignment. We show that our method is able to produce calibrated confidence estimates for predicted root causes, validate the usefulness of retrieved historical data and the prompting strategy as well as the generalizability across different root cause prediction models. Our study takes an important move towards reliably and effectively embedding LLMs into cloud incident management systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
