软件测试和代码共同演化：代码变更后应随之添加新的或更新的测试以记录新的软件行为。然而，现有的测试生成和更新基准通常将测试与代码变更隔离，并依赖静态元数据，无法验证测试是否可执行或是否与代码变更语义关联。这使得评估测试自动化代理是否理解代码变更应如何传播到测试套件变得困难。我们引入TestEvo-Bench，一个从软件仓库挖掘的测试与代码协同演化任务基准，包含两个方向：在测试生成中，代理应编写新测试以捕获新的软件行为；在测试更新中，代理应使现有失败的测试适应变更后的软件行为。每个任务锚定于真实的提交历史，并附带环境配置以支持基于执行的指标，如通过率、覆盖率和变异得分。TestEvo-Bench也是一个动态基准：每个任务记录测试和代码变更的时间戳，新任务通过自动化流水线定期挖掘，因此评估可以限制在模型训练截止日期之后的任务，以减少数据泄露风险。当前快照包含746个测试生成和509个测试更新任务，从152个开源Java项目的59,950个候选协同演化记录中筛选。我们实验了四种结合强大框架（Claude Code、Gemini CLI和SWE-Agent）与强大基础模型（Claude Opus 4.7和Gemini 3.1 Pro）的最先进代理。结果表明，它们在测试生成上达到77.5%的成功率，在测试更新上达到74.6%。然而，在最新的基准任务上成功率显著降低，且在有限的每任务成本下大幅下降。
英文摘要：
Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite.
We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
The Hong Kong Polytechnic University（香港理工大学）；University of Edinburgh（爱丁堡大学）；UCL（伦敦大学学院）；CUHK (SZ)（香港中文大学（深圳））；SUTD（新加坡科技设计大学）；HKUST（香港科技大学）；CUHK（香港中文大学）；HKUST (GZ)（香港科技大学（广州））
VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents
University of Science and Technology of China（中国科学技术大学）；iFLYTEK Research（iFLYTEK研究院）；Xiamen University（厦门大学）；University of Chinese Academy of Sciences（中国科学院大学）
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
University of North Carolina at Chapel Hill（北卡罗来纳大学切里波因特分校）；Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；University of Southern California（南加州大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
Teaching Code LLMs to Reason with Intermediate Formal Specifications
FPT Software AI Center（FPT软件人工智能中心）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；Hanoi Univ. of Science and Tech.（河内科学技术大学）；Texas, USA（得克萨斯州，美国）；Hanoi, Vietnam（河内，越南）
Alibaba Group（阿里巴巴集团）；University of Science and Technology of China（中国科学技术大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
Flower Hub: A Reproducible Benchmarking Platform for Federated Learning in Simulation and Deployment
Flower Labs；University of Cambridge（剑桥大学）；University of Auckland（奥克兰大学）；University of Melbourne（墨尔本大学）；Vector Institute；Fraunhofer IMS（弗劳恩霍夫应用固体物理与材料力学研究所）；NetCompany；Gachon University（嘉泉大学）；Owkin；Sony AI（索尼人工智能）
Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs
Zhejiang University of Technology（浙江工业大学）；Zhejiang Key Laboratory of Visual Information Intelligent Processing（浙江省视觉信息智能处理重点实验室）；University of Aberdeen（阿伯丁大学）；University of Birmingham（伯明翰大学）
GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning
Central South University（中南大学）；Jiangxi Normal University（江西师范大学）；Hohai University（河海大学）；Renmin University of China（中国人民大学）；The Chinese University of Hong Kong（香港中文大学）
SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown
The Hong Kong University of Science and Technology（香港科技大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Harbin Institute of Technology, Harbin（哈尔滨工业大学）
