CloudEval-YAML:面向云配置生成的实用基准
CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation
AI总结:
本文提出首个面向云原生应用的人工编写基准 CloudEval-YAML,包含 1011 个 YAML 配置生成问题及单元测试,并通过可扩展评估平台对 12 个大语言模型进行深入评测,揭示提升性能与降低成本的方法。
摘要:
在云计算生态蓬勃发展和基于大语言模型(LLM)的代码生成工具不断涌现的背景下,云原生应用中的代码生成仍缺乏基准测试。为应对这一需求,我们提出了 CloudEval-YAML,一个面向云配置生成的实用基准。CloudEval-YAML 通过聚焦 YAML 来应对多样性挑战,YAML 是众多云原生工具的事实标准。我们以实用性为导向构建 CloudEval-YAML 基准:数据集由人工编写的问题和针对实际场景的单元测试组成。我们进一步通过以简洁、缩写和双语方式改写问题来增强数据集,以满足实际需求。该数据集包含 1011 个问题,完成这些问题需要超过 1200 个人工小时。为提高评估过程中的实用性,我们为 CloudEval-YAML 构建了一个可扩展的评估平台,其速度比单机评估快 20 倍。据我们所知,CloudEval-YAML 数据集是首个针对云原生应用的人工编写数据集。我们对 12 个大语言模型进行了深入评估,从而对问题和 LLM 有了更深入的理解,并发现了提高任务性能和降低成本的有效方法。
英文摘要:
Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。