In light of the rapid adoption of AI coding assistants, LLM-assisted development has become increasingly prevalent, creating an urgent need for robust evaluation of generated code quality. Existing benchmarks often require extensive manual effort to create static datasets, rely on indirect or insufficiently challenging tasks, depend on non-scalable ground truth, or neglect critical low-level security evaluations, particularly memory-safety issues. In this work, we introduce OSS-Bench, a benchmark generator that automatically constructs large-scale, live evaluation tasks from real-world open-source software. OSS-Bench replaces functions with LLM-generated code and evaluates them using three natural metrics: compilability, functional correctness, and memory safety, leveraging robust signals like compilation failures, test-suite violations, and sanitizer alerts as ground truth. In our evaluation, the benchmark, instantiated as OSS-Bench(php) and OSS-Bench(sql), profiles 17 diverse LLMs, revealing insights such as intra-family behavioral patterns and inconsistencies between model size and performance. Our results demonstrate that OSS-Bench mitigates overfitting by leveraging the evolving complexity of OSS and highlights LLMs' limited understanding of low-level code security via extended fuzzing experiments. Overall, OSS-Bench offers a practical and scalable framework for benchmarking the real-world coding capabilities of LLMs.
Benchmarking Predictive Coding Networks -- Made Simple
University of Oxford（牛津大学）；Vienna University of Technology（维也纳技术大学）；MRC Brain Network Dynamics Unit, University of Oxford（英国医学研究委员会脑网络动力学单元，牛津大学）；VERSES AI Research Lab（VERSES AI 研究实验室）
Decoupling Code Complexity from Newcomer Participation: A Causal Study of AI Coding Agent Adoption in OSS
School of Computer Science, Peking University（北京大学计算机学院）；Key Laboratory of High Confidence Software Technologies, Ministry of Education, China（教育部高可信软件技术重点实验室）
PyGress: Tool for Analyzing the Progression of Code Proficiency in Python OSS Projects
Faculty of Information and Communication Technology, Mahidol University（玛希隆大学信息与通信技术学院）；Graduate School of Information Science and Technology, The University of Osaka（大阪大学信息科学技术研究生院）；Graduate School of Information Science, Nara Institute of Science and Technology（奈良先端科学技术大学院大学信息科学研究生院）
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Beijing Institute of Technology, Beijing, China（北京理工大学）；University of California, Merced, CA, USA（加州大学默塞德分校）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）
SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Shanghai Jiao Tong University（上海交通大学）；Xinjiang University（新疆大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Independent Researcher（独立研究者）；The Chinese University of Hong Kong（香港中文大学）
ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance
NVIDIA（英伟达）；University College London（伦敦大学学院）；New York University（纽约大学）；Georgia Institute of Technology（佐治亚理工学院）；Cornell University（康奈尔大学）；Stanford University（斯坦福大学）；Northeastern University（东北大学）
SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
The University of Hong Kong（香港大学）；Shandong University（山东大学）；Carnegie Mellon University（卡内基梅隆大学）；National University of Singapore（新加坡国立大学）；The Hong Kong University of Science and Technology（香港科技大学）
