In recent times, a plethora of Large Code Generation Models (LCGMs) have been proposed, showcasing significant potential in assisting developers with complex programming tasks. Benchmarking LCGMs necessitates the creation of a set of diverse programming problems, and each problem comprises the prompt (including the task description), canonical solution, and test inputs. The existing methods for constructing such a problem set can be categorized into two main types: manual methods and perturbation-based methods. However, manual methods demand high effort and lack scalability, while also risking data integrity due to LCGMs' potentially contaminated data collection, and perturbation-based approaches mainly generate semantically homogeneous problems with the same canonical solutions and introduce typos that can be easily auto-corrected by IDE, making them ineffective and unrealistic. In this work, we propose the idea of programming problem merging (PPM) and provide two implementation of this idea, we utilize our tool on two widely-used datasets and compare it against nine baseline methods using eight code generation models. The results demonstrate the effectiveness of our tool in generating more challenging, diverse, and natural programming problems, comparing to the baselines.
Benchmarking Language Models for Code Syntax Understanding
University of Maryland, College Park(马里兰大学帕克分校);Google Research(谷歌研究院);Washington University in St. Louis(圣路易斯华盛顿大学);University of California, Berkeley(加州大学伯克利分校)
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Shanghai Jiao Tong University(上海交通大学);Xinjiang University(新疆大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Independent Researcher(独立研究者);The Chinese University of Hong Kong(香港中文大学)
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Key Laboratory of Cyberspace Security, Ministry of Education(教育部网络空间安全重点实验室);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室)
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore(新加坡国立大学);The University of Hong Kong(香港大学);Nanyang Technological University(南洋理工大学);Hong Kong Baptist University(香港浸会大学)
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);ShanghaiTech University(上海科技大学);The University of Hong Kong(香港大学)
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));The Hong Kong University of Science and Technology(香港科学与技术大学);DAMO Academy, Alibaba Group(阿里云达摩院)
QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
New York University (NYU) Abu Dhabi(纽约大学阿布扎比分校);Center for Quantum and Topological Systems (CQTS), NYUAD Research Institute, New York University Abu Dhabi(纽约大学阿布扎比分校NYUAD研究所量子与拓扑系统中心)
Minimalistic Terminal Editor for Julia Programming -- MinTEJ: A Friendly Approach for a Scientific Programmer
Hitachi Energy Technology Services Private Limited(日立能源技术服务私人有限公司);Indian Institute of Technology Bombay(印度孟买理工学院);Newcastle University in Singapore(新加坡纽卡斯尔大学)