Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities.
In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.
DiffuTester: Accelerating Unit Test Generation for Diffusion LLMs via Mining Structural Pattern
College of AI, Tsinghua University（清华大学人工智能学院）；School of Software, Beihang University（北航软件学院）；School of Computer Science and Engineering, Beihang University（北航计算机科学与工程学院）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）
The past few years have seen a surge of applying Deep Learning (DL) models for a wide array of tasks such as image classification, object detection, machine translation, etc. While DL models provide an opportunity to solve otherwise intractable tasks, their adoption relies on them being optimized to meet latency and resource requirements. Benchmarking is a key step in this process but has been hampered in part due to the lack of representative and up-to-date benchmarking suites. This is exacerbated by the fast-evolving pace of DL models. This paper proposes DLBricks, a composable benchmark gen…
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
The Hong Kong Polytechnic University（香港理工大学）；University of Edinburgh（爱丁堡大学）；UCL（伦敦大学学院）；CUHK (SZ)（香港中文大学（深圳））；SUTD（新加坡科技设计大学）；HKUST（香港科技大学）；CUHK（香港中文大学）；HKUST (GZ)（香港科技大学（广州））
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong University of Science and Technology（香港科学与技术大学）；DAMO Academy, Alibaba Group（阿里云达摩院）
QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
New York University (NYU) Abu Dhabi（纽约大学阿布扎比分校）；Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, New York University Abu Dhabi（纽约大学阿布扎比分校NYUAD研究所量子与拓扑系统中心）
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore（新加坡国立大学）；The University of Hong Kong（香港大学）；Nanyang Technological University（南洋理工大学）；Hong Kong Baptist University（香港浸会大学）
New York University（纽约大学）；University of Calgary（卡尔加里大学）
Automating hardware design could obviate a significant amount of human error from the engineering process and lead to fewer errors. Verilog is a popular hardware description language to model and design digital systems, thus generating Verilog code is a critical first step. Emerging large language models (LLMs) are able to write high-quality code in other programming languages. In this paper, we characterize the ability of LLMs to generate useful Verilog. For this, we fine-tune pre-trained LLMs on Verilog datasets collected from GitHub and Verilog textbooks. We construct an evaluation framewor…
