Comprehensive Evaluation and Insights into the Use of Large Language Models in the Automation of Behavior-Driven Development Acceptance Test Formulation
Shanthi Karpurapu, Sravanthy Myneni, Unnati Nettur, Likhit Sagar Gajja, Dave Burke, Tom Stiehm, Jeffery Payne
Behavior-driven development (BDD) is an Agile testing methodology fostering collaboration among developers, QA analysts, and stakeholders. In this manuscript, we propose a novel approach to enhance BDD practices using large language models (LLMs) to automate acceptance test generation. Our study uses zero and few-shot prompts to evaluate LLMs such as GPT-3.5, GPT-4, Llama-2-13B, and PaLM-2. The paper presents a detailed methodology that includes the dataset, prompt techniques, LLMs, and the evaluation process. The results demonstrate that GPT-3.5 and GPT-4 generate error-free BDD acceptance tests with better performance. The few-shot prompt technique highlights its ability to provide higher accuracy by incorporating examples for in-context learning. Furthermore, the study examines syntax errors, validation accuracy, and comparative analysis of LLMs, revealing their effectiveness in enhancing BDD practices. However, our study acknowledges that there are limitations to the proposed approach. We emphasize that this approach can support collaborative BDD processes and create opportunities for future research into automated BDD acceptance test generation using LLMs.
MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs
Department of Computer Science and Engineering, The Chinese University of Hong Kong（中国香港中文大学计算机科学与工程系）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；School of Integrated Circuits, Peking University（北京大学集成电路学院）；School of Intergrated Circuits, Southeast University（东南大学集成电路学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术系）；National Center of Technology Innovation for EDA（EDA技术创新国家中心）
DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice
Product Development Division, Department of Mechanical Engineering, Chalmers University of Technology, Gothenburg, Sweden（产品开发部门，机械工程系，楚姆勒斯技术大学，哥德堡，瑞典）；GKN Aerospace Engine Systems, Trollhättan, Sweden（GKN航空发动机系统，特罗尔哈塔纳，瑞典）
