ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Beijing Institute of Technology, Beijing, China（北京理工大学）；University of California, Merced, CA, USA（加州大学默塞德分校）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）
Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, Ming-Hsuan Yang
Recent coding agents can generate complete codebases from simple prompts, yet existing evaluations focus on issue-level bug fixing and lag behind end-to-end development. We introduce ProjDevBench, an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories. Combining Online Judge (OJ) testing with LLM-assisted code review, the benchmark evaluates agents on (1) system architecture design, (2) functional correctness, and (3) iterative solution refinement. We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios, and evaluate six coding agents built on different LLM backends. Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality and data structures but struggle with complex system design, time complexity optimization, and resource management. Our benchmark is available at https://github.com/zsworld6/projdevbench.
Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging
College of Future Technology and the National Biomedical Imaging Center, Peking University（北京大学未来技术学院和国家生物医学成像中心）；AI for Science Institute (AISI)（科学人工智能研究所）；University of Michigan（密歇根大学）；State Key Laboratory of Acoustics and Marine Information, Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所声场声信息国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Astronautics, Beihang University（北京航空航天大学宇航学院）；Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education（教育部航天器设计优化与动态仿真技术重点实验室）
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
The Chinese University of Hong Kong（香港中文大学）；Independent（独立）；ELLIS；Technical University of Darmstadt（达姆施塔特技术大学）；Johns Hopkins University（约翰霍普金斯大学）；Monash University（墨尔本大学）
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）；Peking University（北京大学）；Southeast University（东南大学）
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Chinese Information Processing Laboratory（中文信息处理实验室）；State Key Laboratory of Computer Science（计算机科学国家重点实验室）；Key Laboratory of System Software, Chinese Academy of Sciences（中国科学院系统软件重点实验室）
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；The Hong Kong University of Science and Technology（香港科学与技术大学）；DAMO Academy, Alibaba Group（阿里云达摩院）
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
FPT Software AI Center（FPT软件人工智能中心）；School of Computing and Information Systems（计算与信息系统学院）；University of Melbourne（墨尔本大学）；Center of AI Research（人工智能研究中心）；VinUniversity（文大学）
CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
New York University Abu Dhabi（纽约大学阿布扎克校区）；Nanyang Technological University（南洋理工大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Harvard University（哈佛大学）；Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University（德雷塞尔大学计算机科学系）；Department of Teaching and Learning, Washington State University（华盛顿州立大学教学与学习系）；College of Chemistry, Beijing Normal University（北京师范大学化学学院）；Department of Computer Science, University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校计算机科学系）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）
FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization
Singapore-MIT Alliance for Research and Technology（新加坡-麻省理工联盟研究技术）；Massachusetts Institute of Technology（麻省理工学院）；Northeastern University（东北大学）；Uber；Shanghai Jiaotong University（上海交通大学）；Boston University（波士顿大学）；Emory University（埃默里大学）；Northwestern University（西北大学）；National University of Singapore（国立新加坡大学）；Microsoft（微软）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Singapore Management University（新加坡管理学院）
AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering
University of Houston（休斯顿大学）；New York University（纽约大学）；Texas A&M University（德克萨斯农工大学）；UCSD（加州大学圣地亚哥分校）；Worcester Polytechnic Institute（伍斯特理工学院）；Oregon State University（俄勒冈州立大学）
