CPP-UT-Bench:LLM 能否用 C++ 编写复杂单元测试?
CPP-UT-Bench: Can LLMs Write Complex Unit Tests in C++?
AI总结:
本文构建涵盖 14 个开源 C++ 代码库、2,653 个代码-测试对的 CPP-UT-Bench,并通过上下文学习、PEFT 与全参数微调评估 LLM 的复杂 C++ 单元测试生成能力。
摘要:
我们提出 CPP-UT-Bench,这是一个用于衡量大型语言模型(LLM)的 C++ 单元测试生成能力的基准数据集。CPP-UT-Bench 旨在反映现实世界中广泛而多样的 C++ 代码库集合。该数据集包含 2,653 个 {代码,单元测试} 对,这些样本来自 14 个不同的开源 C++ 代码库,横跨九个不同领域,包括机器学习、软件测试、解析、标准输入输出、数据工程、日志、完整表达式求值、键值存储和服务器协议。我们通过在上下文学习、参数高效微调(PEFT)和全参数微调方面的大量实验,证明了 CPP-UT-Bench 作为基准数据集的有效性。我们还讨论了数据集编译的挑战,以及从上下文学习和微调实验中获得的洞见。除 CPP-UT-Bench 数据集和数据编译代码外,我们还提供微调后的模型权重,以供进一步研究。在十项实验中的九项里,我们的微调 LLM 相比相应基础模型平均提升超过 70%。
英文摘要:
We introduce CPP-UT-Bench, a benchmark dataset to measure C++ unit test generation capability of a large language model (LLM). CPP-UT-Bench aims to reflect a broad and diverse set of C++ codebases found in the real world. The dataset includes 2,653 {code, unit test} pairs drawn from 14 different opensource C++ codebases spanned across nine diverse domains including machine learning, software testing, parsing, standard input-output, data engineering, logging, complete expression evaluation, key value storage, and server protocols. We demonstrated the effectiveness of CPP-UT-Bench as a benchmark dataset through extensive experiments in in-context learning, parameter-efficient fine-tuning (PEFT), and full-parameter fine-tuning. We also discussed the challenges of the dataset compilation and insights we learned from in-context learning and fine-tuning experiments. Besides the CPP-UT-Bench dataset and data compilation code, we are also offering the fine-tuned model weights for further research. For nine out of ten experiments, our fine-tuned LLMs outperformed the corresponding base models by an average of more than 70%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。