代码大模型 / AI 编程

Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

Yixin Cao, Shibo Hong, Xinze Li, Jiahao Ying, Yubo Ma, Haiyuan Liang, Yantao Liu, Zijun Yao, Xiaozhi Wang, Dan Huang, Wenxuan Zhang, Lifu Huang, Muhao Chen, Lei Hou, Qianru Sun, Xingjun Ma, Zuxuan Wu, Min-Yen Kan, David Lo, Qi Zhang, Heng Ji, Jing Jiang, Juanzi Li, Aixin Sun, Xuanjing Huang, Tat-Seng Chua, Yu-Gang Jiang

机构 * Fudan University（复旦大学） ； Nanyang Technological University（南洋理工大学） ； Singapore Management University（新加坡管理学院） ； Tsinghua University（清华大学） ； Singapore University of Technology and Design（新加坡科技设计大学） ； University of California Davis（加州大学戴维斯分校） ； National University of Singapore（新加坡国立大学） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Australian National University（澳大利亚国立大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

Representation Learning for Tabular Data: A Comprehensive Survey

Jun-Peng Jiang, Si-Yang Liu, Hao-Run Cai, Qile Zhou, Han-Jia Ye

机构 * School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学） ； National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学） 纠错

专题命中 代码评测 ：repository(abstract)；分类 cs.LG

BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models

Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, Fei Yuan

机构 * National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学） ； University of Zurich（苏黎世大学） ； Carnegie Mellon University（卡内基梅隆大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） 纠错

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL

Benchmarking LLM-based Relevance Judgment Methods

Negar Arabzadeh, Charles L. A. Clarke

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, Amelia Glaese

专题命中 代码评测 ：coding agent(abstract)；分类 cs.CL

TurtleBench: A Visual Programming Benchmark in Turtle Geometry

Sina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi

专题命中 代码评测 ：code generation(abstract)；分类 cs.AI

Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding

Pedro Hermosilla, Christian Stippel, Leon Sick

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

Comments Accepted at CVPR 2025

CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs

Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T. Doan, Nam V. Nguyen, Quang Pham, Nghi D. Q. Bui

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE

ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback

Tasnia Rahman, Sathish A. P. Kumar, Sumit Jha, Arvind Ramanathan

专题命中 代码评测 ：program repair(abstract)；分类 cs.LG

Comments 9 pages, 3 figures

Does RAG Introduce Unfairness in LLMs? Evaluating Fairness in Retrieval-Augmented Generation Systems

Xuyang Wu, Shuowei Li, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

Comments Published at COLING 2025

Fixseeker: An Empirical Driven Graph-based Approach for Detecting Silent Vulnerability Fixes in Open Source Software

Yiran Cheng, Ting Zhang, Lwin Khin Shar, Zhe Lang, David Lo, Shichao Lv, Dongliang Fang, Zhiqiang Shi, Limin Sun

专题命中 代码评测 ：repository(abstract)；分类 cs.SE

Software Testing for Extended Reality Applications: A Systematic Mapping Study

Ruizhen Gu, José Miguel Rojas, Donghwan Shin

专题命中 代码评测 ：repository(abstract)；分类 cs.SE

Comments 51 pages, 10 figures

Domain Adaptation for Japanese Sentence Embeddings with Contrastive Learning based on Synthetic Sentence Generation

Zihao Chen, Hisashi Handa, Miho Ohsaki, Kimiaki Shirahama

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

Comments 39 pages, 7 figures

ClimDetect: A Benchmark Dataset for Climate Change Detection and Attribution

Sungduk Yu, Brian L. White, Anahita Bhiwandiwalla, Musashi Hinck, Matthew Lyle Olson, Yaniv Gurwicz, Raanan Y. Rohekar, Tung Nguyen, Vasudev Lal

专题命中 代码评测 ：repository(abstract)；分类 cs.LG

LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding

Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin

专题命中 代码评测 ：code model(abstract)；分类 cs.SE

Comments 18 pages

VirtualXAI: A User-Centric Framework for Explainability Assessment Leveraging GPT-Generated Personas

Georgios Makridis, Vasileios Koukos, Georgios Fatouros, Dimosthenis Kyriazis

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

Comments 8 pages, 6 figures

Counterfactual Learning on Graphs: A Survey

Zhimeng Guo, Teng Xiao, Zongyu Wu, Charu Aggarwal, Hui Liu, Suhang Wang

专题命中 代码评测 ：repository(abstract)；分类 cs.LG

Journal ref Machine Intelligence Research (vol. 22, no. 1, pp. 17-59), https://link.springer.com/article/10.1007/s11633-024-1519-z, 2025

CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models

Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, Yuanxing Zhang, Zhexu Wang, Weixun Wang, Yancheng He, Ken Deng, Wangchunshu Zhou, Wenhao Huang, Zhaoxiang Zhang

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL

A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models

Zhongzhan Huang, Shanshan Zhong, Pan Zhou, Shanghua Gao, Marinka Zitnik, Liang Lin

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

Comments Accepted by TPAMI. arXiv admin note: text overlap with arXiv:2312.02439

How to Get Your LLM to Generate Challenging Problems for Evaluation

Arkil Patel, Siva Reddy, Dzmitry Bahdanau

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

Robust Feature Engineering Techniques for Designing Efficient Motor Imagery-Based BCI-Systems

Syed Saim Gardezi, Soyiba Jawed, Mahnoor Khan, Muneeba Bukhari, Rizwan Ahmed Khan

专题命中 代码评测 ：repository(abstract)；分类 cs.LG

Comments 26 pages

How well can a large language model explain business processes as perceived by users?

Dirk Fahland, Fabiana Fournier, Lior Limonad, Inna Skarbovsky, Ava J. E. Swevels

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

Comments 42 pages, 13 figures

Journal ref Data & Knowledge Engineering, Volume 157, May 2025

Performance Evaluation of Large Language Models in Statistical Programming

Xinyi Song, Kexin Xie, Lina Lee, Ruizhe Chen, Jared M. Clark, Hao He, Haoran He, Jie Min, Xinlei Zhang, Simin Zheng, Zhiyang Zhang, Xinwei Deng, Yili Hong

专题命中 代码评测 ：code generation(abstract)；分类 cs.AI

Comments 27 pages, 8 figures

LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights

Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, Jeff Huang

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

Comments 33 pages, 12 figures

HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks

Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, Jacky Keung

专题命中 代码评测 ：code generation(abstract)；分类 cs.AI

Comments homepage https://humaneval-v.github.io/

Copilot Arena: A Platform for Code LLM Evaluation in the Wild

Wayne Chi, Valerie Chen, Anastasios Nikolas Angelopoulos, Wei-Lin Chiang, Aditya Mittal, Naman Jain, Tianjun Zhang, Ion Stoica, Chris Donahue, Ameet Talwalkar

专题命中 代码评测 ：code generation(abstract)；分类 cs.SE

MetRex: A Benchmark for Verilog Code Metric Reasoning Using LLMs

Manar Abdelatty, Jingxiao Ma, Sherief Reda

专题命中 代码评测 ：code generation(abstract)；分类 cs.CL

WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages

Jia Yu, Fei Yuan, Rui Min, Jing Yu, Pei Chu, Jiayang Li, Wei Li, Ruijie Zhang, Zhenxiang Li, Zhifei Ren, Dong Zheng, Wenjian Zhang, Yan Teng, Lingyu Meng, ZhenJiang Jin, Jiantao Qiu, ShaSha Wang, Zhongying Tu, Dahua Lin, Yu Wang, Yu Qiao, Yanfeng Wang, Conghui He

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

CLOFAI: A Dataset of Real And Fake Image Classification Tasks for Continual Learning

William Doherty, Anton Lee, Heitor Murilo Gomes

专题命中 代码评测 ：repository(abstract)；分类 cs.AI

SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editing

Varun Biyyala, Bharat Chanderprakash Kathuria, Jialu Li, Youshan Zhang

专题命中 代码评测 ：repository(abstract)；分类 cs.CL

Comments WACV workshop