Context: The rapid evolution of Large Language Models (LLMs) has sparked significant interest in leveraging their capabilities for automating code review processes. Prior studies often focus on developing LLMs for code review automation, yet require expensive resources, which is infeasible for organizations with limited budgets and resources. Thus, fine-tuning and prompt engineering are the two common approaches to leveraging LLMs for code review automation. Objective: We aim to investigate the performance of LLMs-based code review automation based on two contexts, i.e., when LLMs are leveraged by fine-tuning and prompting. Fine-tuning involves training the model on a specific code review dataset, while prompting involves providing explicit instructions to guide the model's generation process without requiring a specific code review dataset. Method: We leverage model fine-tuning and inference techniques (i.e., zero-shot learning, few-shot learning and persona) on LLMs-based code review automation. In total, we investigate 12 variations of two LLMs-based code review automation (i.e., GPT- 3.5 and Magicoder), and compare them with the Guo et al.'s approach and three existing code review automation approaches. Results: The fine-tuning of GPT 3.5 with zero-shot learning helps GPT-3.5 to achieve 73.17% -74.23% higher EM than the Guo et al.'s approach. In addition, when GPT-3.5 is not fine-tuned, GPT-3.5 with few-shot learning achieves 46.38% - 659.09% higher EM than GPT-3.5 with zero-shot learning. Conclusions: Based on our results, we recommend that (1) LLMs for code review automation should be fine-tuned to achieve the highest performance; and (2) when data is not sufficient for model fine-tuning (e.g., a cold-start problem), few-shot learning without a persona should be used for LLMs for code review automation.
From Prompting to Engineering: A Research Agenda for Prompt Engineering in Software Engineering
Universitat Politècnica de Catalunya(加泰罗尼亚理工大学);CNR(意大利国家研究委员会);Pegaso University(佩加索大学);University of Gothenburg and Chalmers University of Technology(哥德堡大学与查尔姆斯理工大学);Politecnico di Torino(都灵理工大学);University of Glasgow(格拉斯哥大学);Tampere University(坦佩雷大学)
DUCTILE: Agentic LLM Orchestration of Engineering Analysis in Product Development Practice
Product Development Division, Department of Mechanical Engineering, Chalmers University of Technology, Gothenburg, Sweden(产品开发部门,机械工程系,楚姆勒斯技术大学,哥德堡,瑞典);GKN Aerospace Engine Systems, Trollhättan, Sweden(GKN航空发动机系统,特罗尔哈塔纳,瑞典)
Comprehensive Evaluation and Insights into the Use of Large Language Models in the Automation of Behavior-Driven Development Acceptance Test Formulation
When Prompt Engineering Meets Software Engineering: CNL-P as Natural and Robust "APIs'' for Human-AI Interaction
CSIRO’s Data61(CSIRO数据61实验室);Jiangxi Provincial Key Laboratory for High Performance Computing(江西省高性能计算重点实验室);State International Science & Technology Cooperation Base of Networked Supporting Software(网络化支持软件国际科技合作基地);Language Intelligence Research Center(语言智能研究中心);Jiangxi Normal University(江西师范大学);Institute of AI, Newcastle University, UK(人工智能研究院,新castle大学,英国);UGAiForge
Prompt Engineering Strategies for LLM-based Qualitative Coding of Psychological Safety in Software Engineering Communities: A Controlled Empirical Study
Federal University of Bahia(巴伊亚联邦大学);State University of Feira de Santana(费拉德桑塔纳州立大学);Federal University of Sergipe(塞格皮联邦大学);Federal Institute of Ceara(塞阿拉联邦理工学院)
Challenges and Paths Towards AI for Software Engineering
MIT CSAIL(麻省理工学院计算机科学与人工智能实验室);University of California, Berkeley(加州大学伯克利分校);Cornell University(康奈尔大学);Stanford University(斯坦福大学);University of Pennsylvania(宾夕法尼亚大学)