arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.00409cs.CRcs.AIcs.LGcs.SE
收藏

DiverseVul:一个面向基于深度学习的漏洞检测的新型漏洞源代码数据集

DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection

University of Maryland(马里兰大学);UC Berkeley(加州大学伯克利分校);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科技城);Google Deepmind(谷歌DeepMind)

Yizheng Chen, Zhoujie Ding, Lamya Alowain, Xinyun Chen, David Wagner

AI总结:

本文发布包含18,945个漏洞函数和330,492个非漏洞函数的DiverseVul数据集,并通过11种模型实验表明深度学习在漏洞检测上仍面临高假阳性、低F1和泛化困难等挑战,同时指出LLM和源代码特定预训练是更有前景的方向。

摘要:

我们提出并发布了一个新的漏洞源代码数据集。我们通过爬取安全议题网站,从相应项目中提取漏洞修复提交和源代码来整理该数据集。我们的新数据集包含18,945个易受攻击的函数,涵盖150个CWE,以及从7,514次提交中提取的330,492个非易受攻击的函数。我们的数据集覆盖的项目数量比以往所有数据集的总和还多295个。将我们的新数据集与以往数据集相结合,我们对使用深度学习检测软件漏洞所面临的挑战和有前景的研究方向进行了分析。我们研究了属于4个家族的11种模型架构。我们的结果表明,由于假阳性率高、F1分数低以及难以检测困难的CWE,深度学习仍未准备好用于漏洞检测。特别是,我们展示了基于深度学习的模型部署所面临的一个重要泛化挑战。我们表明,增加训练数据量可能不会进一步提高深度学习模型在漏洞检测上的性能,但可能有助于提高对未见项目的泛化能力。我们还确定了有希望的未来研究方向。我们证明,大型语言模型(LLMs)是基于机器学习的漏洞检测的一个有前景的研究方向,在我们的实验中,其性能优于具有代码结构特征的图神经网络(GNNs)。此外,开发源代码特定的预训练目标是一个有前景的研究方向,可以提高漏洞检测性能。

英文摘要:

We propose and release a new vulnerable source code dataset. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits. Our dataset covers 295 more projects than all previous datasets combined. Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities. We study 11 model architectures belonging to 4 families. Our results show that deep learning is still not ready for vulnerability detection, due to high false positive rate, low F1 score, and difficulty of detecting hard CWEs. In particular, we demonstrate an important generalization challenge for the deployment of deep learning-based models. We show that increasing the volume of training data may not further improve the performance of deep learning models for vulnerability detection, but might be useful to improve the generalization ability to unseen projects. We also identify hopeful future research directions. We demonstrate that large language models (LLMs) are a promising research direction for ML-based vulnerability detection, outperforming Graph Neural Networks (GNNs) with code-structure features in our experiments. Moreover, developing source code specific pre-training objectives is a promising research direction to improve the vulnerability detection performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑