代码大模型 / AI 编程

MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, Bryan Hooi

机构 * National University of Singapore（新加坡国立大学） ； University of California, Santa Barbara（加州大学圣巴巴拉分校） ； Singapore University of Technology and Design（新加坡科技设计大学） 纠错

专题命中 代码评测 ：coding agent(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 49 pages, 9 figures. Accepted by NeurIPS 2025 D&B Track

Learning to Reason as Action Abstractions with Scalable Mid-Training RL

Shenao Zhang, Donghan Yu, Yihao Feng, Bowen Jin, Zhaoran Wang, John Peebles, Zirui Wang

机构 * Apple（苹果公司） ； Northwestern University（西北大学） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning

Alexander Golubev, Maria Trofimova, Sergei Polezhaev, Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Sergey Abramov, Andrei Andriushchenko, Filipp Fisin, Sergei Skvortsov, Boris Yangel

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.LG

HealthFlow: A Self-Evolving AI Agent with Meta Planning for Autonomous Healthcare Research

Yinghao Zhu, Yifan Qi, Zixiang Wang, Lei Gu, Dehao Sui, Haoran Hu, Xichen Zhang, Ziyi He, Junjun He, Liantao Ma, Lequan Yu

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code: https://github.com/yhzhu99/HealthFlow

Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation

Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, Baishakhi Ray

机构 * Columbia University（哥伦比亚大学） ； National University of Singapore（国立新加坡大学） ； University of California, Riverside（加州大学河滨分校） ； University of Southern California（南加州大学） ； The Ohio State University（俄亥俄州立大学） ； Peking University（北京大学） ； Tsinghua University（清华大学） ； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Github Link: https://github.com/SeekingDream/Static-to-Dynamic-LLMEval

Intuition to Evidence: Measuring AI's True Impact on Developer Productivity

Anand Kumar, Vishal Khare, Deepak Sharma, Satyam Kumar, Vijay Saini, Anshul Yadav, Sachendra Jain, Ankit Rana, Pratham Verma, Vaibhav Meena, Avinash Edubilli

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.LG

Comments 16 pages, 10 figures, 5 tables

HSFN: Hierarchical Selection for Fake News Detection building Heterogeneous Ensemble

Sara B. Coutinho, Rafael M. O. Cruz, Francimaria R. S. Nascimento, George D. C. Cavalcanti

机构 * Centro de Informática (CIn), Universidade Federal de Pernambuco (UFPE)（计算机学院（CIn），佛罗里达州立大学（UFPE）） ； École de Technologie Supérieure (ÉTS), Université du Québec（高级技术学院（ÉTS），魁北克大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by IEEE International Conference on Systems, Man, and Cybernetics (SMC) - IEEE SMC 2025

MetaGen: A DSL, Database, and Benchmark for VLM-Assisted Metamaterial Generation

Liane Makatura, Benjamin Jones, Siyuan Bian, Wojciech Matusik

机构 * MIT CSAIL（麻省理工学院计算机科学与人工智能实验室） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG、cs.PL

SABER: Switchable and Balanced Training for Efficient LLM Reasoning

Kai Zhao, Yanjun Zhao, Jiaming Song, Shien He, Lusheng Zhang, Qiang Zhang, Tianjiao Li

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Geak: Introducing Triton Kernel AI Agent & Evaluation Benchmarks

Jianghui Wang, Vinay Joshi, Saptarshi Majumder, Xu Chao, Bin Ding, Ziqiong Liu, Pratik Prabhanjan Brahma, Dong Li, Zicheng Liu, Emad Barsoum

机构 * Advanced Micro Devices, Inc.（先进微器件公司） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?

Muntasir Wahed, Xiaona Zhou, Kiet A. Nguyen, Tianjiao Yu, Nirav Diwan, Gang Wang, Dilek Hakkani-Tür, Ismini Lourentzou

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Winner Defender Team at Amazon Nova AI Challenge 2025

CodeMirage: Hallucinations in Code Generated by Large Language Models

Vibhor Agarwal, Yulong Pei, Salwa Alamir, Xiaomo Liu

机构 * University of Surrey（萨里大学） ； JP Morgan AI Research（摩根大通AI研究） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments Accepted at AutoMates @ IJCAI 2024

MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark

Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, Furu Wei

机构 * Microsoft Research（微软研究院） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

Isaac Chung, Imene Kerboua, Marton Kardos, Roman Solomatin, Kenneth Enevoldsen

机构 * ITMO University（ITMO大学） ； Aarhus University（奥胡斯大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.CL、cs.AI

SWE-bench Goes Live!

Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, Dongmei Zhang

机构 * Microsoft（微软） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments Homepage: \url{https://swe-bench-live.github.io/}, Code: \url{https://github.com/SWE-bench-Live}, Dataset: \url{https://huggingface.co/SWE-bench-Live}

CityGPT: Empowering Urban Spatial Cognition of Large Language Models

Jie Feng, Tianhui Liu, Yuwei Du, Siqi Guo, Yuming Lin, Yong Li

机构 * Department of Electronic Engineering, BNRist, Tsinghua University（电子工程系、BNRist、清华大学） ； School of Electronic and Information Engineering, Beijing Jiaotong University（电子信息工程学院、北京交通大学） ； Department of Electronic Engineering, Tsinghua University（电子工程系、清华大学） ； Department of Urban Planning, Tsinghua University（城市规划系、清华大学） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by KDD 2025 Research Track, https://github.com/tsinghua-fib-lab/CityGPT

BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluation

Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel, Amit Agarwal, Alice Oh

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP

Hanxun Huang, Sarah Erfani, Yige Li, Xingjun Ma, James Bailey

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

Can Large Language Models Understand Symbolic Graphics Programs?

Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf

机构 * Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所） ； University of Cambridge（剑桥大学） ； MIT（麻省理工学院） 纠错

专题命中 代码评测 ：program synthesis(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2025 Spotlight (v4: 47 pages, 26 figures, project page: https://sgp-bench.github.io/)

ACECODER: Acing Coder RL via Automated Test-Case Synthesis

Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, Wenhu Chen

专题命中 代码评测 ：code model(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments 9 pages, 4 figure, 11 tables. Accepted to ACL 2025 main conference

Causal Cartographer: From Mapping to Reasoning Over Counterfactual Worlds

Gaël Gendron, Jože M. Rožanec, Michael Witbrock, Gillian Dobbie

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 29 pages, 9 pages for the main paper, 20 pages for the references and appendix, 25 figures

AI-Mediated Code Comment Improvement

Maria Dhakal, Chia-Yi Su, Robert Wallace, Chris Fakhimi, Aakash Bansal, Toby Li, Yu Huang, Collin McMillan

机构 * Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，诺丁汉大学） ； Division of Computer Science and Engineering, Louisiana State University（计算机科学与工程分校，路易斯安那州立大学） ； Department of Computer Science, Vanderbilt University（计算机科学系，范德比大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.AI、cs.PL

Open Llama2 Model for the Lithuanian Language

Artūras Nakvosas, Povilas Daniušis, Vytas Mulevičius

机构 * Neurotechnology（神经技术） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 12 pages, 8 figures, 5 tables

Journal ref Informatica, 2025

Themisto: Jupyter-Based Runtime Benchmark

Konstantin Grotov, Sergey Titov

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.LG

Comments Accepted to the third Deep Learning for Code (DL4C) workshop @ ICLR 2025

VeriMind: Agentic LLM for Automated Verilog Generation with a Novel Evaluation Metric

Bardia Nadimi, Ghali Omar Boutaib, Hao Zheng

专题命中 代码评测 ：code generation(abstract)；分类 cs.AI、cs.LG、cs.PL

A System for Comprehensive Assessment of RAG Frameworks

Mattia Rengo, Senad Beadini, Domenico Alfano, Roberto Abbruzzese

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Technical Report, 7 pages, 2 figures, 1 table

ResBench: Benchmarking LLM-Generated FPGA Designs with Resource Awareness

Ce Guo, Tong Zhao

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments to be published in International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies 2025

The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination

Yifan Sun, Han Wang, Dongbai Li, Gang Wang, Huan Zhang

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 23 pages

Can LLMs Enable Verification in Mainstream Programming?

Aleksandr Shefer, Igor Engel, Stanislav Alekseev, Daniil Berezun, Ekaterina Verbitskaia, Anton Podkopaev

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.PL

Automatically Labeling Clinical Trial Outcomes: A Large-Scale Benchmark for Drug Development

Chufan Gao, Jathurshan Pradeepkumar, Trisha Das, Shivashankar Thati, Jimeng Sun

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG