arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.06786cs.DCcs.AI
收藏

CloudEval-YAML:面向云配置生成的实用基准

CloudEval-YAML: A Practical Benchmark for Cloud Configuration Generation

Alibaba Cloud(阿里云);University of Michigan(密歇根大学);UCLA(加州大学洛杉矶分校);UC Merced(加州大学默塞德分校)

Yifei Xu, Yuning Chen, Xumiao Zhang, Xianshang Lin, Pan Hu, Yunfei Ma, Songwu Lu, Wan Du, Zhuoqing Mao, Ennan Zhai, Dennis Cai

AI总结:

本文提出首个面向云原生应用的人工编写基准 CloudEval-YAML,包含 1011 个 YAML 配置生成问题及单元测试,并通过可扩展评估平台对 12 个大语言模型进行深入评测,揭示提升性能与降低成本的方法。

摘要:

在云计算生态蓬勃发展和基于大语言模型(LLM)的代码生成工具不断涌现的背景下,云原生应用中的代码生成仍缺乏基准测试。为应对这一需求,我们提出了 CloudEval-YAML,一个面向云配置生成的实用基准。CloudEval-YAML 通过聚焦 YAML 来应对多样性挑战,YAML 是众多云原生工具的事实标准。我们以实用性为导向构建 CloudEval-YAML 基准:数据集由人工编写的问题和针对实际场景的单元测试组成。我们进一步通过以简洁、缩写和双语方式改写问题来增强数据集,以满足实际需求。该数据集包含 1011 个问题,完成这些问题需要超过 1200 个人工小时。为提高评估过程中的实用性,我们为 CloudEval-YAML 构建了一个可扩展的评估平台,其速度比单机评估快 20 倍。据我们所知,CloudEval-YAML 数据集是首个针对云原生应用的人工编写数据集。我们对 12 个大语言模型进行了深入评估,从而对问题和 LLM 有了更深入的理解,并发现了提高任务性能和降低成本的有效方法。

英文摘要:

Among the thriving ecosystem of cloud computing and the proliferation of Large Language Model (LLM)-based code generation tools, there is a lack of benchmarking for code generation in cloud-native applications. In response to this need, we present CloudEval-YAML, a practical benchmark for cloud configuration generation. CloudEval-YAML tackles the diversity challenge by focusing on YAML, the de facto standard of numerous cloud-native tools. We develop the CloudEval-YAML benchmark with practicality in mind: the dataset consists of hand-written problems with unit tests targeting practical scenarios. We further enhanced the dataset to meet practical needs by rephrasing questions in a concise, abbreviated, and bilingual manner. The dataset consists of 1011 problems that take more than 1200 human hours to complete. To improve practicality during evaluation, we build a scalable evaluation platform for CloudEval-YAML that achieves a 20 times speedup over a single machine. To the best of our knowledge, the CloudEval-YAML dataset is the first hand-written dataset targeting cloud-native applications. We present an in-depth evaluation of 12 LLMs, leading to a deeper understanding of the problems and LLMs, as well as effective methods to improve task performance and reduce cost.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑