代码大模型 / AI 编程

Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition

Kehua Feng, Keyan Ding, Hongzhi Tan, Kede Ma, Zhihua Wang, Shuangquan Guo, Yuzhou Cheng, Ge Sun, Guozhou Zheng, Qiang Zhang, Huajun Chen

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.LG

Comments 35 pages, 6 figures, Accepted by ACL 2025

THiNK: Can Large Language Models Think-aloud?

Yongan Yu, Mengqian Wu, Yiran Lin, Nikki G. Lobczowski

机构 * McGill University（麦吉尔大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation

Guang Yang, Yu Zhou, Xiang Chen, Wei Zheng, Xing Hu, Xin Zhou, David Lo, Taolue Chen

机构 * College of Computer Science and Technology（计算机科学与技术学院） ； School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院） ； School of Software（软件学院） ； School of Software Technology（软件技术学院） ； School of Computing and Information Systems（计算与信息系统学院） ； School of Computing and Mathematical Sciences（计算与数学科学学院） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI

Large Language Models for Predictive Analysis: How Far Are They?

Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi

机构 * Peking University（北京大学） ； Harvard University（哈佛大学） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025 Findings

Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, Shafiq Joty

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025. The first two authors contributed equally. The codebase is at https://github.com/SalesforceAIResearch/jetts-benchmark

Benchmarking Post-Training Quantization in LLMs: Comprehensive Taxonomy, Unified Evaluation, and Comparative Analysis

Jiaqi Zhao, Ming Wang, Miao Zhang, Yuzhang Shang, Xuebo Liu, Yaowei Wang, Min Zhang, Liqiang Nie

机构 * Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）） ； Illinois Institute of Technology（伊利诺伊理工学院） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Comments 17 pages, 3 fugures

A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics

Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu, Maksym Ziemlewski, Paris Loizides, Sander Vermeulen, Arie van Deursen, Maliheh Izadi

机构 * Delft University of Technology（代尔夫特理工大学） 纠错

专题命中 代码评测 ：code model(abstract)；分类 cs.SE、cs.AI

Comments Accepted PROMISE '25

EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization

Dong Huang, Jianbo Dai, Han Weng, Puzhen Wu, Yuhao Qing, Heming Cui, Zhijiang Guo, Jie M. Zhang

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL

Comments Accepted by NeurIPS 2024

EffiBench: Benchmarking the Efficiency of Automatically Generated Code

Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, Jie M. Zhang

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL

Comments Camera Ready for NeurIPS 2024

English Please: Evaluating Machine Translation with Large Language Models for Multilingual Bug Reports

Avinash Patil, Siru Tao, Aryan Jadon

机构 * Juniper Networks Inc.（Juniper网络公司） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.CL

Comments 8 Pages, 4 Figures, 3 Tables

Ensuring Reproducibility in Generative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets

Masumi Morishige, Ryo Koshihara

机构 * Galirage Inc.（加利拉格公司） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 10 figures

CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

Wenjing Yin, Tianze Sun, Yijiong Yu, Jiawei Fang, Guangyao Su, Jiancheng Wang, Zekun Wang, Wei Wang, Ran Chen, Ziyun Dai, Shuai Yuan, Menghang Dong, Peng Luo, Dong Cao, Da Lei, Yajun Zhang, Hao Chen, Xiang Ma, Yong Liu, Weifeng Liu, Yuanjian Xu, Ji Pei

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI

Comments Submitted to ACL 2025. Under review

Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks

Kang Yang, Xinjun Mao, Shangwen Wang, Yanlin Wang, Tanghaoran Zhang, Bo Lin, Yihao Qin, Zhang Zhang, Yao Lu, Kamal Al-Sabahi

机构 * College of Computer, National University of Defense Technology（计算机学院，国防科技大学） ； Sun Yat-sen University（中山大学） ； College of Banking and Financial Studies（金融学院） 纠错

专题命中 代码评测 ：code model(abstract)；分类 cs.SE、cs.CL

Comments Awarded the ACM SIGSOFT Distinguished Paper Award in ICPC 2025

Gradient-Optimized Fuzzy Classifier: A Benchmark Study Against State-of-the-Art Models

Magnus Sieverding, Nathan Steffen, Kelly Cohen

机构 * University of Cincinnati（辛辛那提大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, Xin Xia

机构 * Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳）） ； Monash University Malaysia（墨尔本大学马来西亚分校） ； Zhejiang University（浙江大学） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI

Comments Accepted by ISSTA 2025: https://conf.researchr.org/details/issta-2025/issta-2025-papers/85/Can-LLMs-replace-Human-Evaluators-An-Empirical-Study-of-LLM-as-a-Judge-in-Software-E

RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins

Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, Lunkai Lin, Zhiqiang Xie, Mingyu Ding, Ping Luo

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI

Comments CVPR 2025 Highlight. 22 pages. Project page: https://robotwin-benchmark.github.io/

RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)

Yao Mu, Tianxing Chen, Shijia Peng, Zanxin Chen, Zeyu Gao, Yude Zou, Lunkai Lin, Zhiqiang Xie, Ping Luo

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI

Comments Project page: https://robotwin-benchmark.github.io/early-version/

OpenTuringBench: An Open-Model-based Benchmark and Framework for Machine-Generated Text Detection and Attribution

Lucio La Cava, Andrea Tagarelli

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

Comments Under review with ARR

Agent-Arena: A General Framework for Evaluating Control Algorithms

Halid Abdulrahim Kadi, Kasim Terzić

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.AI

Comments 20 pages and 1 figure

Your Image Generator Is Your New Private Dataset

Nicolo Resmini, Eugenio Lomurno, Cristian Sbrolli, Matteo Matteucci

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Uncertainty Propagation in XAI: A Comparison of Analytical and Empirical Estimators

Teodor Chiaburu, Felix Bießmann, Frank Haußer

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Comments 23 pages, 10 figures, accepted at WCXAI 2025 Istanbul

FAIRE: Assessing Racial and Gender Bias in AI-Driven Resume Evaluations

Athena Wen, Tanush Patil, Ansh Saxena, Yicheng Fu, Sean O'Brien, Kevin Zhu

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

TS-Inverse: A Gradient Inversion Attack Tailored for Federated Time Series Forecasting Models

Caspar Meijer, Jiyue Huang, Shreshtha Sharma, Elena Lazovik, Lydia Y. Chen

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

EnvBench: A Benchmark for Automated Environment Setup

Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, Yaroslav Zharov

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.LG

Comments Accepted at the DL4Code workshop at ICLR'25

Job Shop Scheduling Benchmark: Environments and Instances for Learning and Non-learning Methods

Robbert Reijnen, Igor G. Smit, Hongxiang Zhang, Yaoxin Wu, Zaharah Bukhsh, Yingqian Zhang

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models

Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, Hua Wei

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Comments 19 pages, 6 figures, 5 tables

Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model

Wenke Huang, Jian Liang, Xianda Guo, Yiyang Fang, Guancheng Wan, Xuankun Rong, Chi Wen, Zekun Shi, Qingyun Li, Didi Zhu, Yanbiao Ma, Ke Liang, Bin Yang, He Li, Jiawei Shao, Mang Ye, Bo Du

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity

Zahra Gharaee, Scott C. Lowe, ZeMing Gong, Pablo Millan Arias, Nicholas Pellegrino, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Lila Kari, Dirk Steinke, Graham W. Taylor, Paul Fieguth, Angel X. Chang

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

Journal ref NeurIPS 2024

Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis

Jiahao Gai, Hao Mark Chen, Zhican Wang, Hongyu Zhou, Wanru Zhao, Nicholas Lane, Hongxiang Fan

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.LG

Comments Paper accepted by ASP-DAC'25

CoCoNUT: Structural Code Understanding does not fall out of a tree

Claas Beger, Saikat Dutta

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.LG

Comments Accepted at 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code)