arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2611.06373cs.SE
收藏

重新评估PrimeVul数据集上漏洞检测的数据集质量与微调实践

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset

University of Szeged(塞格德大学);FrontEndART Ltd.(FrontEndART有限公司)

Norbert Sándor Szolnoki, Gábor Antal

AI总结:

本文针对PrimeVul漏洞检测场景,对CodeBERT等模型开展大规模微调评估,发现现有指标存在偏差,PrimeVul数据集存在结构性问题,需改进数据集设计与评估及微调实践。

摘要:

近期研究报告称,在PrimeVul等基准上,对预训练代码语言模型进行监督微调可在自动漏洞检测任务中取得良好结果。本文针对该场景下的模型性能与数据集质量开展大规模实证评估,系统地对三种广泛使用的编码器模型(CodeBERT、GraphCodeBERT和UniXCoder)进行微调,覆盖五种不平衡缓解策略、大量超参数搜索及多个随机种子,共完成770次独立训练运行。研究结果揭示了一致模式:尽管F1、ROC-AUC等标准指标偶尔显示性能提升,但安全关键型及校准感知型指标呈现不同结果——马修斯相关系数接近零,预测概率校准效果差,在严格误报约束下评估显示高误报率;不平衡处理、阈值优化及事后校准均无法恢复有意义的判别信号。此外,本文对PrimeVul基准进行数据质量评估,识别出结构性问题,如漏洞函数与修复函数间语义差异极小、提交级标注引入的标注噪声。这些发现表明,当前形式的PrimeVul数据集对软件工程中基于AI的可靠漏洞检测构成重大挑战,本研究强调需设计更稳健的数据集、改进评估协议并重新思考安全关键型任务的微调实践。

英文摘要:

Recent studies report promising results from supervised fine-tuning of pre-trained code language models for automated vulnerability detection on benchmarks such as PrimeVul. In this paper, we present a large-scale empirical evaluation of both model performance and dataset quality in this setting. We systematically fine-tune three widely used encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) across five imbalance mitigation strategies, extensive hyperparameter sweeps, and multiple random seeds, resulting in 770 independent training runs. Our results reveal a consistent pattern. While standard metrics such as F1 and ROC-AUC occasionally suggest improvements, security-critical and calibration-aware measures tell a different story. Matthews Correlation Coefficient remains near zero, predicted probabilities are poorly calibrated, and evaluation under strict false-positive constraints reveals high false-negative rates. Neither imbalance handling, threshold optimization, nor post-hoc calibration recovers a meaningful discriminative signal. We also conduct a data quality assessment of the PrimeVul benchmark and identify structural issues such as minimal semantic differences between vulnerable and fixed functions, and annotation noise introduced by commit-level labeling. These findings suggests that the PrimeVul dataset, in its current form, presents significant challenges for reliable AI-based vulnerability detection in software engineering with the methodologies we used. Our study highlights the need for more robust dataset design, improved evaluation protocols, and rethinking of fine-tuning practices for security-critical tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑