利用大型语言模型支持CI/CD流水线故障管理的自动化：SAP HANA案例研究
Using Large Language Models to Support Automation of Failure Management in CI/CD Pipelines: A Case Study in SAP HANA
AI总结：
本研究利用大型语言模型结合历史故障数据，实现了CI/CD流水线故障管理的自动化，准确率达92.1%。
摘要：
在手动执行CI/CD流水线故障管理时，过程耗费时间。自动化这一过程具有挑战性，因为有效故障管理所需的信息是无结构化的，无法被传统程序自动处理。凭借处理无结构化数据的能力，大型语言模型（LLMs）已在之前的研究所中展示了自动化故障管理的潜力。在这些研究之后，我们评估了基于LLM的系统是否能够在大型工业软件项目（即SAP HANA）的CI/CD流水线中实现自动化故障管理。我们评估了LLM系统识别错误位置和提出精确解决方案的能力，这些解决方案不包含不必要的操作。为了支持LLM生成精确解决方案，我们为其提供了不同类型的领域知识，包括流水线信息、故障管理指南和历史故障数据。我们进行了消融研究，以确定哪种类型的领域知识对解决方案的准确性贡献最大。结果表明，历史故障数据对系统准确性贡献最大，使系统在我们的数据集中能够产生精确的解决方案，在92.1%的情况下。当提供领域知识时，系统在识别错误位置的准确性为97.4%，而没有提供时为84.2%。结论是，我们的发现表明，当提供历史故障数据时，LLMs在自动化CI/CD流水线故障管理方面代表了一种有前景的方法。
英文摘要：
CI/CD pipeline failure management is time-consuming when performed manually. Automating this process is non-trivial because the information required for effective failure management is unstructured and cannot be automatically processed by traditional programs. With their ability to process unstructured data, large language models (LLMs) have shown promising results for automated failure management by previous work. Following these studies, we evaluated whether an LLM-based system could automate failure management in a CI/CD pipeline in the context of a large industrial software project, namely SAP HANA. We evaluated the ability of the LLM-based system to identify the error location and to propose exact solutions that contain no unnecessary actions. To support the LLM in generating exact solutions, we provided it with different types of domain knowledge, including pipeline information, failure management instructions, and data from historical failures. We conducted an ablation study to determine which type of domain knowledge contributed most to solution accuracy. The results show that data from historical failures contributed the most to the system's accuracy, enabling it to produce exact solutions in 92.1% of cases in our dataset. The system correctly identified the error location with 97.4% accuracy when provided with domain knowledge, compared to 84.2% accuracy without it. In conclusion, our findings indicate that LLMs, when provided with data from historical failures, represent a promising approach for automating CI/CD pipeline failure management.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
