Recent advancements in large language models (LLMs) have automated various software engineering tasks, with benchmarks emerging to evaluate their capabilities. However, for adaptation, a critical activity during code reuse, there is no benchmark to assess LLMs' performance, leaving their practical utility in this area unclear. To fill this gap, we propose AdaptEval, a benchmark designed to evaluate LLMs on code snippet adaptation. Unlike existing benchmarks, AdaptEval incorporates the following three distinctive features: First, Practical Context. Tasks in AdaptEval are derived from developers' practices, preserving rich contextual information from Stack Overflow and GitHub communities. Second, Multi-granularity Annotation. Each task is annotated with requirements at both task and adaptation levels, supporting the evaluation of LLMs across diverse adaptation scenarios. Third, Fine-grained Evaluation. AdaptEval includes a two-tier testing framework combining adaptation-level and function-level tests, which enables evaluating LLMs' performance across various individual adaptations. Based on AdaptEval, we conduct the first empirical study to evaluate six instruction-tuned LLMs and especially three reasoning LLMs on code snippet adaptation. Experimental results demonstrate that AdaptEval enables the assessment of LLMs' adaptation capabilities from various perspectives. It also provides critical insights into their current limitations, particularly their struggle to follow explicit instructions. We hope AdaptEval can facilitate further investigation and enhancement of LLMs' capabilities in code snippet adaptation, supporting their real-world applications.
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
Tencent（腾讯公司）；Peking University（北京大学）；Fudan University（复旦大学）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Singapore Management University（新加坡国立大学）
I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution
University of Central Florida（中佛罗里达大学）；Nanyang Technological University（南洋理工大学）；Technical University of Munich（慕尼黑工业大学）；Propulsion and Space Research Center, Technology Innovation Institute（技术创新研究所推进与空间研究中心）
CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detection
Eötvös Loránd University (ELTE)（厄特沃什·罗兰大学（ELTE））；University of Oslo（奥斯陆大学）；Cyentific AS；Guelma University（盖勒马大学）；The University of Manchester（曼彻斯特大学）；Federal University of Amazonas（亚马逊联邦大学）；Technology Innovation Institute (TII)（技术创新研究院（TII））
CODE-DITING: A Reasoning-Based Metric for Functional Alignment in Code Evaluation
College of Computer Science and Technology（计算机科学与技术学院）；School of Artificial Intelligence and Computer Science（人工智能与计算机科学学院）；School of Software（软件学院）；School of Software Technology（软件技术学院）；School of Computing and Information Systems（计算与信息系统学院）；School of Computing and Mathematical Sciences（计算与数学科学学院）
Citekit: A Modular Toolkit for Large Language Model Citation Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
RedCode: Risky Code Execution and Generation Benchmark for Code Agents
University of Chicago（芝加哥大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Lapis Labs（拉皮斯实验室）；Virginia Tech（弗吉尼亚理工大学）；Microsoft Research（微软研究院）；University of California Berkeley（加州大学伯克利分校）
