代码大模型 / AI 编程

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

Li Zhong, Zilong Wang, Jingbo Shang

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments Preprint

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation

Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, Shuiguang Deng

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments Accepted by ACL 2024 main conference

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.LG

Comments Website - https://livecodebench.github.io/

Kotlin ML Pack: Technical Report

Sergey Titov, Mikhail Evtikhiev, Anton Shapkin, Oleg Smirnov, Sergei Boytsov, Sergei Boytsov, Dariia Karaeva, Maksim Sheptyakov, Mikhail Arkhipov, Timofey Bryksin, Egor Bogomolov

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.PL

Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers

Elijah Pelofske, Vincent Urias, Lorie M. Liebrock

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

LeTI: Learning to Generate from Textual Interactions

Xingyao Wang, Hao Peng, Reyhaneh Jabbarvand, Heng Ji

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments NAACL 2024 Findings

ICE-Score: Instructing Large Language Models to Evaluate Code

Terry Yue Zhuo

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.AI

Comments Accepted to Findings of EACL 2024

CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model

Peng Di, Jianguo Li, Hang Yu, Wei Jiang, Wenting Cai, Yang Cao, Chaoyu Chen, Dajun Chen, Hongwei Chen, Liang Chen, Gang Fan, Jie Gong, Zi Gong, Wen Hu, Tingting Guo, Zhichao Lei, Ting Li, Zheng Li, Ming Liang, Cong Liao, Bingchang Liu, Jiachen Liu, Zhiwei Liu, Shaojun Lu, Min Shen, Guangpei Wang, Huan Wang, Zhi Wang, Zhaogui Xu, Jiawei Yang, Qing Ye, Gehao Zhang, Yu Zhang, Zelin Zhao, Xunjin Zheng, Hailian Zhou, Lifu Zhu, Xianying Zhu

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.LG

Comments Accepted by ICSE-SEIP 2024

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution

Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, Sida I. Wang

专题命中 代码评测 ：code model(abstract)；分类 cs.SE、cs.AI、cs.LG

Comments 71 pages, 29 figures

Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data

Yu Yang, Aaditya K. Singh, Mostafa Elhoushi, Anas Mahmoud, Kushal Tirumala, Fabian Gloeckle, Baptiste Rozière, Carole-Jean Wu, Ari S. Morcos, Newsha Ardalani

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 12 pages, 4 figures, Oral Presentation at 3rd Workshop on Efficient Natural Language and Speech Processing (ENLSP-III), NeurIPS 2023

Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review

Mingze Yuan, Peng Bao, Jiajia Yuan, Yunhao Shen, Zifan Chen, Yi Xie, Jie Zhao, Yang Chen, Li Zhang, Lin Shen, Bin Dong

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 1 figure, 3 tables

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.CL、cs.LG

Comments Project site with code and data: https://intercode-benchmark.github.io

Spanish Pre-trained BERT Model and Evaluation Data

José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge Pérez

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published as workshop paper at Practical ML for Developing Countries Workshop @ ICLR 2020

A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets

Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Xiangji Huang

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ACL 2023 Findings. The first three authors contributed equally

Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions

Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, Nick Haber

专题命中 代码评测 ：program synthesis(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments humaneval results, clarity

Measuring The Impact Of Programming Language Distribution

Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Malmaud, Jacob Austin, Rishabh Singh, Michele Catasta

专题命中 代码评测 ：code model(abstract)；分类 cs.CL、cs.LG、cs.PL

Comments Accepted to ICML 2023, Code and data release: https://github.com/google-research/babelcode

Fault-Aware Neural Code Rankers

Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu K Lahiri, Madanlal Musuvathi, Jianfeng Gao

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE、cs.AI、cs.PL

Comments In the proceedings of Advances in Neural Information Processing Systems, 2022

Towards Using Data-Influence Methods to Detect Noisy Samples in Source Code Corpora

Anh T. V. Dau, Thang Nguyen-Duc, Hoang Thanh-Tung, Nghi D. Q. Bui

专题命中 代码评测 ：code model(abstract)；分类 cs.SE、cs.AI、cs.PL

Comments The 37th IEEE/ACM International Conference on Automated Software Engineering

Depression Symptoms Modelling from Social Media Text: A Semi-supervised Learning Approach

Nawshad Farruque, Randy Goebel, Sudhakar Sivapalan, Osmar Zaiane

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Title and relevant changes are made

C-Pack of IPAs: A C90 Program Benchmark of Introductory Programming Assignments

Pedro Orvalho, Mikoláš Janota, Vasco Manquinho

专题命中 代码评测 ：program repair(abstract)；分类 cs.SE、cs.AI、cs.PL

Comments 3 pages, 3 tables, 1 GitHub url: https://github.com/pmorvalho/C-Pack-IPAs

SapientML: Synthesizing Machine Learning Pipelines by Learning from Human-Written Solutions

Ripon K. Saha, Akira Ura, Sonal Mahajan, Chenguang Zhu, Linyi Li, Yang Hu, Hiroaki Yoshida, Sarfraz Khurshid, Mukul R. Prasad

专题命中 代码评测 ：program synthesis(abstract)；分类 cs.SE、cs.AI、cs.LG

Comments Accepted to the Technical Track of ICSE 2022

Curriculum-Meta Learning for Order-Robust Continual Relation Extraction

Tongtong Wu, Xuekai Li, Yuan-Fang Li, Reza Haffari, Guilin Qi, Yujin Zhu, Guoqiang Xu

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by AAAI2021

WizardCoder: Empowering Code Large Language Models with Evol-Instruct

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, Daxin Jiang

机构 * Microsoft（微软公司） ； Hong Kong Baptist University（香港 Baptist 大学） 纠错

专题命中 代码评测 ：code generation(abstract,comments)；分类 cs.CL、cs.AI

Comments Large Language model, Code Generation, Code LLMs.This paper has been accepted to ICLR 2024. Please cite the ICLR version

Journal ref The Twelfth International Conference on Learning Representations (ICLR 2024)

Can GPT-O1 Kill All Bugs? An Evaluation of GPT-Family LLMs on QuixBugs

Haichuan Hu, Ye Shang, Guolin Xu, Congqing He, Quanjun Zhang

专题命中 代码评测 ：program repair(abstract,comments)；分类 cs.SE、cs.AI

Comments Accepted to the 6th International Workshop on Automated Program Repair (APR 2025)

INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers

Anjan Karmakar, Romain Robbes

专题命中 代码评测 ：code model(abstract,comments)；分类 cs.SE、cs.LG

Comments Accepted to IEEE Transactions on Software Engineering. Extension of our previous paper "What do pre-trained code models know about code?" (ASE 2021, arXiv:2108.11308). 21 pages

Automating Multi-Hop RAG Evaluation via TRIAD: From Context Extraction to Validated Dataset Generation

通过TRIAD实现多跳RAG评估自动化：从上下文提取到验证数据集生成

Lorenz Brehme, Adam Jatowt

机构 * University of Innsbruck（因斯布鲁克大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

AI总结 本文提出TRIAD三阶段自动化多跳RAG评估数据集生成方法，经与MuSiQue、HotpotQA对比验证，该生成数据集可有效评估特定领域RAG系统性能，相关代码与验证结果已公开。

Comments Accepted at the 19th International Natural Language Generation Conference (INLG 2026)

Runtime Action Interference for AI Control of AlphaStar in StarCraft II

星际争霸II中AlphaStar的AI控制之运行时动作干预

Jaymari Chua, Chen Wang, Liming Zhu, Lina Yao

机构 * University of New South Wales（新南威尔士大学） ； CSIRO（澳大利亚联邦科学与工业研究组织） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.AI、cs.LG

AI总结 该研究提出运行时动作干预（RAI）机制，将其应用于AlphaStar复现版并开展《星际争霸II》人类实验，发现向用户披露AI对手能力会显著影响人类对其公平性、毒性的感知，需将执行栈控制与能力披露分开评估。

Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap

超越基准：基于拟人化与生命周期导向路线图的大语言模型评估

Jun Wang, Ninglun Gu, Kailai Zhang, Pengyong Li, Yelun Bao, Jin Yang, Xu Yin, Liwei Liu, Zijiao Zhang, Yihuan Liu, Gary G. Yen, Junchi Yan

机构 * Department of Networks, China Mobile Communications Group Co.,Ltd.（中国移动通信集团有限公司网络部） ； Xidian University（西安电子科技大学） ； Oklahoma State University（俄克拉荷马州立大学） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL、cs.AI

AI总结 本研究针对LLM基准分数与实际效用脱节问题，建立诊断本体并提出含IQ、PQ、EQ、VQ的拟人化评估框架，分析200余个基准后为LLM开发提供战略指引。

Comments Preprint. Under Review

Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda

大型语言模型在软件工程与软件安全交叉领域：一项以证据为中心的结构化调查与研究议程

Wei Lin, Tao Zhou, Zhaofei Xie, Changgui Hong

机构 * Nanjing Liancheng Intelligent Technology Group（南京连城智能科技集团） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.SE、cs.AI

AI总结 本研究调查了LLMs在软件工程与软件安全交叉领域的进展，提出保证框架，识别有效性威胁与最低报告协议，制定研究议程，主张以任务适配证据而非单一基准评判模型能力。

TokEval: A Tokenizer Evaluation Suite

TokEval：一个分词器评估套件

Clara Meister

机构 * EPFL（洛桑联邦理工学院） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL、cs.LG

AI总结 本研究推出TokEval框架，通过信息论、结构敏感等指标评估分词器，经实验验证其可预测下游模型性能，助力更原则性的分词器评估。

Comments Published as a conference paper at COLM 2026; Library hosted at https://github.com/cimeister/tokenizer-intrinsic-evals