arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.04729cs.SE
收藏

基于大语言模型的行为驱动开发场景生成

Behaviour Driven Development Scenario Generation with Large Language Models

School of Computing Technologies, RMIT University(RMIT大学计算技术学院)

Amila Rathnayake, Mojtaba Shahin, Golnoush Abaei

AI总结:

本文研究了三种大语言模型在BDD场景生成中的表现,发现Claude 3在人类和模型评估中表现最佳,提示技术和输入质量对生成质量有显著影响。

摘要:

本文评估了三种大语言模型(GPT-4、Claude 3和Gemini)在自动化行为驱动开发(BDD)场景生成中的表现。为了支持这一评估,我们构建了一个包含500个用户故事、需求描述及其相应BDD场景的数据集,这些数据来自四个专有软件产品。我们通过包含文本和语义相似性度量、基于大语言模型的评估和人类专家评估的多维评估框架,评估了LLM生成的BDD场景质量。我们的发现表明,尽管GPT-4在文本和语义相似性度量上得分更高,但Claude 3生成的场景在人类专家和基于LLM的评估者中评分最高。基于LLM的评估者,尤其是DeepSeek,比文本相似性和语义相似性度量更与人类判断相关。提示技术的有效性是模型特定的:GPT-4在零样本情况下表现最佳,Claude 3受益于链式推理,而Gemini在少量示例情况下表现最优。输入质量决定了BDD场景生成的有效性:详细的请求描述单独产生高质量的场景,而用户故事单独产生低质量的场景。我们的实验表明,设置温度为0和top_p为1.0在所有模型中产生了最高质量的BDD场景。

英文摘要:

This paper presents an evaluation of three LLMs, GPT-4, Claude 3, and Gemini, for automated Behaviour-Driven Development (BDD) scenarios generation. To support this evaluation, we constructed a dataset of 500 user stories, requirement descriptions, and their corresponding BDD scenarios, drawn from four proprietary software products. We assessed the quality of BDD scenarios generated by LLMs using a multidimensional evaluation framework encompassing text and semantic similarity metrics, LLM-based evaluation, and human expert assessment. Our findings reveal that although GPT-4 achieves higher scores in text and semantic similarity metrics, Claude 3 produces scenarios rated highest by both human experts and LLM-based evaluators. LLM-based evaluators, particularly DeepSeek, show a stronger correlation with human judgment than with text similarity and semantic similarity metrics. The effectiveness of prompting techniques is model-specific: GPT-4 performs best with zero-shot, Claude 3 benefits from chain-of-thought reasoning, and Gemini achieves optimal results with few-shot examples. Input quality determines the effectiveness of BDD scenario generation: detailed requirement descriptions alone yield high-quality scenarios, whereas user stories alone yield low-quality scenarios. Our experiments indicate that setting temperature to 0 and top_p to 1.0 produced the highest-quality BDD scenarios across all models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑