Large language models (LLMs) have demonstrated impressive capabilities across a wide range of coding tasks, including summarization, translation, completion, and code generation. Despite these advances, detecting code vulnerabilities remains a challenging problem for LLMs. In-context learning (ICL) has emerged as an effective mechanism for improving model performance by providing a small number of labeled examples within the prompt. Prior work has shown, however, that the effectiveness of ICL depends critically on how these few-shot examples are selected. In this paper, we study two intuitive criteria for selecting few-shot examples for ICL in the context of code vulnerability detection. The first criterion leverages model behavior by prioritizing samples on which the LLM consistently makes mistakes, motivated by the intuition that such samples can expose and correct systematic model weaknesses. The second criterion selects examples based on semantic similarity to the query program, using k-nearest-neighbor retrieval to identify relevant contexts.
We conduct extensive evaluations using open-source LLMs and datasets spanning multiple programming languages. Our results show that for Python and JavaScript, careful selection of few-shot examples can lead to measurable performance improvements in vulnerability detection. In contrast, for C and C++ programs, few-shot example selection has limited impact, suggesting that more powerful but also more expensive approaches, such as re-training or fine-tuning, may be required to substantially improve model performance.
Automated Vulnerability Injection in Smart Contracts Using Large Language Models
Università degli Studi di Napoli Federico II(那不勒斯费德里科二世大学);Gran Sasso Science Institute (GSSI)(格兰萨索科学研究所);University of Coimbra(科英布拉大学);College of Computing and Informatics, University of North Carolina at Charlotte(北卡罗来纳大学夏洛特分校计算与信息学院)
VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models
Institute of BigData, Fudan University(复旦大学大数据研究院);School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science, Huazhong University of Science and Technology(华中科技大学计算机科学学院);Purple Mountain Laboratories(紫金山实验室);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院)