Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Tsinghua University（清华大学）；Peking University（北京大学）；Southeast University（东南大学）
Agents driven by large language models are increasingly deployed on long-horizon tasks, where they use tools and modify real workspaces. How well they perform depends as much on the harness around the model as on the model itself. Existing benchmarks measure downstream task success under a fixed harness, leaving harness engineering itself unexplored. We propose Doc2Harness, a two-stage benchmark that evaluates long-horizon harness engineering directly. In Stage 1, a build model operating inside a fixed builder harness receives an interface-only contract skeleton and a development document that consolidates behavior across twelve frontier harnesses, and then constructs a complete candidate harness in an isolated container. In Stage 2, the frozen candidate harness runs in containers separate from the construction environment, one per test case, and is driven end to end by a scripted mock model. We score each test case using four observable channels, namely the requests sent to the model, the files on disk, the process exit code, and the command output; positive controls rule out vacuous passes. Across nineteen frontier models, each evaluated with three independent builds on 328 test cases, GPT-6 Astra achieves the highest pass rate at 53.05%, while the weakest model reaches 27.44%, a gap of roughly twenty-six percentage points. Even the strongest model fails nearly half of the test cases, and failures cluster around how cross-cutting functionality is split across modules. To our knowledge, Doc2Harness provides the first deterministic benchmark for measuring long-horizon harness engineering.
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Beijing Institute of Technology, Beijing, China（北京理工大学）；University of California, Merced, CA, USA（加州大学默塞德分校）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）
Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study
School of Information Technologies Tallinn University of Technology（塔林理工大学信息技术学院）；School of Informatics, Computing, and Cyber Systems Northern Arizona University（北亚利桑那大学信息学、计算与网络系统学院）
Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging
College of Future Technology and the National Biomedical Imaging Center, Peking University（北京大学未来技术学院和国家生物医学成像中心）；AI for Science Institute (AISI)（科学人工智能研究所）；University of Michigan（密歇根大学）；State Key Laboratory of Acoustics and Marine Information, Institute of Acoustics, Chinese Academy of Sciences（中国科学院声学研究所声场声信息国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Astronautics, Beihang University（北京航空航天大学宇航学院）；Key Laboratory of Spacecraft Design Optimization and Dynamic Simulation Technologies, Ministry of Education（教育部航天器设计优化与动态仿真技术重点实验室）
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
The Chinese University of Hong Kong（香港中文大学）；Independent（独立）；ELLIS；Technical University of Darmstadt（达姆施塔特技术大学）；Johns Hopkins University（约翰霍普金斯大学）；Monash University（墨尔本大学）
HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses?
School of Software Engineering, Sun Yat-sen University（中山大学软件学院）；Peng Cheng Laboratory（鹏城实验室）；East China Normal University（华东师范大学）；Department of Computer Science, Hong Kong Baptist University（香港浸会大学计算机系）；Tsinghua University（清华大学）；Department of New Networks, Peng Cheng Laboratory（鹏城实验室新网络部）
SpecGuardBench: Can Coding Agents Construct Reliable Validation Harnesses from Natural Language Specifications?
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Aalto University（阿尔托大学）；Lero, the Research Ireland Centre for Software（Lero——爱尔兰软件研究中心）；University of Limerick（利默里克大学）；University of Ottawa（渥太华大学）
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
FPT Software AI Center（FPT软件人工智能中心）；School of Computing and Information Systems（计算与信息系统学院）；University of Melbourne（墨尔本大学）；Center of AI Research（人工智能研究中心）；VinUniversity（文大学）
CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
New York University Abu Dhabi（纽约大学阿布扎克校区）；Nanyang Technological University（南洋理工大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Harvard University（哈佛大学）；Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing University of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）
