基于执行的Bash和PowerShell自然语言到代码生成用于事件修复的评估
Execution-Based Evaluation of Natural Language to Bash and PowerShell for Incident Remediation
AI总结：
本文提出首个基于执行的评估平台，含125个手工测试用例，评估LLMs生成的Bash和PowerShell代码，并基准测试七个开源与闭源模型，以自动修复APM发现的系统事件。
摘要：
鉴于大型语言模型（LLMs）的最新进展，代码生成任务因其在不同领域的广泛应用而受到极大关注。为了评估并选择最佳模型来自动修复由应用性能监控（APM）平台发现的系统事件，验证生成的代码在语法和语义上是否正确，以及能否按预期正确执行至关重要。然而，当前评估LLMs生成代码质量的方法严重依赖表面形式相似性度量（如BLEU、ROUGE和精确/部分匹配），这些方法存在诸多局限性。相比之下，基于执行的评估更侧重于代码功能，并且不将代码生成限制于任何固定解决方案。然而，设计和实现这样的基于执行的评估平台并非易事。已有若干工作为SQL、Python、Java等流行编程语言创建了基于执行的评估平台，但对于Bash和PowerShell等脚本语言的尝试有限甚至没有。在本文中，我们提出了首个基于执行的评估平台，其中我们创建了三个测试套件（共125个手工测试用例），用于评估LLMs生成的Bash（包括单行命令和多行脚本）和PowerShell代码。我们使用我们的平台，采用不同技术（零样本与少样本学习）对七个闭源和开源LLMs进行了基准测试。
英文摘要：
Given recent advancements of Large Language Models (LLMs), code generation tasks attract immense attention for wide application in different domains. In an effort to evaluate and select a best model to automatically remediate system incidents discovered by Application Performance Monitoring (APM) platforms, it is crucial to verify if the generated code is syntactically and semantically correct, and whether it can be executed correctly as intended. However, current methods for evaluating the quality of code generated by LLMs heavily rely on surface form similarity metrics (e.g. BLEU, ROUGE, and exact/partial match) which have numerous limitations. In contrast, execution based evaluation focuses more on code functionality and does not constrain the code generation to any fixed solution. Nevertheless, designing and implementing such execution-based evaluation platform is not a trivial task. There are several works creating execution-based evaluation platforms for popular programming languages such as SQL, Python, Java, but limited or no attempts for scripting languages such as Bash and PowerShell. In this paper, we present the first execution-based evaluation platform in which we created three test suites (total 125 handcrafted test cases) to evaluate Bash (both single-line commands and multiple-line scripts) and PowerShell codes generated by LLMs. We benchmark seven closed and open-source LLMs using our platform with different techniques (zero-shot vs. few-shot learning).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
