Code-focused Large Language Models (LLMs), such as CodeX and Star-Coder, have demonstrated remarkable capabilities in enhancing developer productivity through context-aware code generation. However, evaluating the quality and security of LLM-generated code remains a significant challenge. Existing evaluation protocols for Code LLMs lack both methodological rigor and comprehensive scope. A key limitation is dataset bias, which arises from unintentional overlap between training and testing data. Furthermore, while CodeBLEU, a BLEU-based metric, is widely used to assess code similarity, it suffers from critical shortcomings, including imprecise tokenization, structural limitations, and low reference diversity. To address these challenges, we introduce CFCEval, a novel framework for evaluating the quality and security of code generated by LLMs. CFCEval mitigates dataset bias by creating a new benchmark, MLVBench, and incorporates ELRM, a new metric designed to assess the relevance between reference code and generated code. CFCEval evaluates generated code across four dimensions: programming quality, vulnerability-fixing capability, post-transformation fixing capability, and relevance. Our experiments show that CFCEval not only captures both quality and security aspects of generated code more effectively but also that its ELRM aligns more closely with human judgments than CodeBLEU, thus paving the way for future advancements in Code LLMs evaluation.
A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code
Tencent(腾讯公司);Peking University(北京大学);Fudan University(复旦大学);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Zhejiang University(浙江大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Singapore Management University(新加坡国立大学)
Evaluating and Improving the Robustness of Security Attack Detectors Generated by LLMs
Samuele Pasini Universit\`a della Svizzera italiana, Switzerland Jinhan Kim Universit\`a della Svizzera italiana, Switzerland Tommaso Aiello SAP Labs France, France Rocio Cabrera Lozoya SAP Labs France, France Antonino Sabetta SAP Labs France, France Paolo Tonella Universit\`a della Svizzera italiana, Switzerland
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
Sun Yat-sen University(中山大学);Nanyang Technological University(南洋理工大学);Alibaba Group(阿里巴巴集团);The State Key Laboratory of Blockchain and Data Security, Zhejiang University(浙江大学区块链与数据安全国家重点实验室)