The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings.
UCSF（加州大学旧金山分校）；SFVAMC（旧金山退伍军人事务医疗中心）；UC Berkeley（加州大学伯克利分校）；Helen Diller Family Comprehensive Cancer Center, UCSF（加州大学旧金山分校海伦·迪勒家族综合癌症中心）；Bakar Computational Health Sciences Institute, UCSF（加州大学旧金山分校巴卡尔计算健康科学研究所）
Vulnerability Detection with Code Language Models: How Far Are We?
University of Maryland（马里兰大学）；King Abdulaziz City for Science and Technology（阿卜杜勒阿齐兹国王科学技术城）；Google DeepMind（谷歌DeepMind）；Columbia University（哥伦比亚大学）；University of Washington（华盛顿大学）
Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
Singapore Management University（新加坡管理大学）；Monash University（蒙纳士大学）；Chinese University of Hong Kong（香港中文大学）；University of Groningen（格罗宁根大学）；Wuhan University（武汉大学）；GovTech（政府科技局）
