arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.06248cs.SE
收藏

CFCEval: 评估大型语言模型生成代码的安全性方面

CFCEval: Evaluating Security Aspects in Code Generated by Large Language Models

Concordia University(康考迪亚大学)

Cheng Cheng, Jinqiu Yang

AI总结:

CFCEval通过引入MLVBench和ELRM指标,有效评估大型语言模型生成代码的质量与安全性,提升代码评估的准确性和全面性。

摘要:

针对代码聚焦的大型语言模型(如CodeX和Star-Coder)在通过上下文感知代码生成提升开发者生产力方面表现出的显著能力,评估生成代码的质量和安全性仍是一个重大挑战。现有的Code LLMs评估协议在方法论严谨性和全面性方面都存在不足。一个关键限制是数据集偏差,这源于训练数据和测试数据之间无意的重叠。此外,虽然基于BLEU的CodeBLEU指标被广泛用于评估代码相似性,但它存在关键缺陷,包括不精确的分词、结构限制和低参考多样性。为了解决这些挑战,我们引入了CFCEval,一种用于评估大型语言模型生成代码质量和安全性的新框架。CFCEval通过创建一个新的基准MLVBench来减轻数据集偏差,并纳入ELRM,一个新指标,用于评估参考代码和生成代码之间的相关性。CFCEval在四个维度上评估生成代码:编程质量、漏洞修复能力、后转换修复能力和相关性。我们的实验表明,CFCEval不仅更有效地捕捉生成代码的质量和安全性方面,而且其ELRM比CodeBLEU更贴近人类判断,从而为未来Code LLMs评估的进步铺平了道路。

英文摘要:

Code-focused Large Language Models (LLMs), such as CodeX and Star-Coder, have demonstrated remarkable capabilities in enhancing developer productivity through context-aware code generation. However, evaluating the quality and security of LLM-generated code remains a significant challenge. Existing evaluation protocols for Code LLMs lack both methodological rigor and comprehensive scope. A key limitation is dataset bias, which arises from unintentional overlap between training and testing data. Furthermore, while CodeBLEU, a BLEU-based metric, is widely used to assess code similarity, it suffers from critical shortcomings, including imprecise tokenization, structural limitations, and low reference diversity. To address these challenges, we introduce CFCEval, a novel framework for evaluating the quality and security of code generated by LLMs. CFCEval mitigates dataset bias by creating a new benchmark, MLVBench, and incorporates ELRM, a new metric designed to assess the relevance between reference code and generated code. CFCEval evaluates generated code across four dimensions: programming quality, vulnerability-fixing capability, post-transformation fixing capability, and relevance. Our experiments show that CFCEval not only captures both quality and security aspects of generated code more effectively but also that its ELRM aligns more closely with human judgments than CodeBLEU, thus paving the way for future advancements in Code LLMs evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑