Test-driven development (TDD) is the practice of writing tests first and coding later, and the proponents of TDD expound its numerous benefits. For instance, given an issue on a source code repository, tests can clarify the desired behavior among stake-holders before anyone writes code for the agreed-upon fix. Although there has been a lot of work on automated test generation for the practice "write code first, test later", there has been little such automation for TDD. Ideally, tests for TDD should be fail-to-pass (i.e., fail before the issue is resolved and pass after) and have good adequacy with respect to covering the code changed during issue resolution. This paper introduces TDD-Bench Verified, a high-quality benchmark suite of 449 issues mined from real-world GitHub code repositories. The benchmark's evaluation harness runs only relevant tests in isolation for simple yet accurate coverage measurements, and the benchmark's dataset is filtered both by human judges and by execution in the harness. This paper also presents Auto-TDD, an LLM-based solution that takes as input an issue description and a codebase (prior to issue resolution) and returns as output a test that can be used to validate the changes made for resolving the issue. Our evaluation shows that Auto-TDD yields a better fail-to-pass rate than the strongest prior work while also yielding high coverage adequacy. Overall, we hope that this work helps make developers more productive at resolving issues while simultaneously leading to more robust fixes.
RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));INSAIT;Sofia University “St. Kliment Ohridski”(索非亚大学“圣·克利门特·奥赫里德斯基”);Shenzhen Loop Area Institute(深圳河套学院);AIRS
Vero: Can AI Agents Build Formally Verified Software Repositories?
University of Chicago(芝加哥大学);California Institute of Technology(加州理工学院);Stanford University(斯坦福大学);UC Berkeley(加州大学伯克利分校);Amazon Web Services(亚马逊云计算服务);Apodex
TDD-Agent: Test-Driven Reasoning for Code Generation
National College for Excellent Engineers, Beihang University(北京航空航天大学卓越工程师学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机科学与工程学院);Qingdao Research Institute and Hangzhou Innovation Institute, Beihang University(北京航空航天大学青岛研究院与杭州创新研究院);School of Software, Beihang University(北京航空航天大学软件学院)
Lero, the Research Ireland Centre for Software, University of Limerick(利默里克大学 Lero 爱尔兰软件研究中心);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Hong Kong University of Science and Technology(香港科技大学);University of Ottawa(渥太华大学)
BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills
Cornell University(康奈尔大学);University of California San Diego(加州大学圣地亚哥分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Microsoft Research(微软研究院)
Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
School of Software, Beihang University, Beijing, China(北京航空航天大学软件学院);Shanghai AI Laboratory, Shanghai, China(上海人工智能实验室);Shanghai Innovation Institute, Shanghai, China(上海创新研究院)