arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.27675cs.SEcs.CRcs.LG
收藏

在有效基于LLM的漏洞检测中选择少样本示例的难度

On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection

Colorado State University(科罗拉多州立大学);Carnegie Mellon University(卡内基梅隆大学)

Md Abdul Hannan, Ronghao Ni, Chi Zhang, Limin Jia, Ravi Mangal, Corina S. Pasareanu

AI总结:

本文研究了在基于LLM的漏洞检测中选择少样本示例的两种标准,并发现不同编程语言对示例选择的响应不同。

摘要:

大型语言模型(LLMs)在广泛编码任务中展现了令人印象深刻的能力,包括摘要、翻译、完成和代码生成。尽管取得了这些进展,但检测代码漏洞仍然是LLMs面临的挑战问题。上下文学习(ICL)已出现作为提高模型性能的有效机制,通过在提示中提供少量标记示例。然而,先前的工作表明,ICL的效果在很大程度上取决于这些少样本示例如何选择。在本文中,我们研究了两种直观的标准,用于在代码漏洞检测的上下文中选择ICL的少样本示例。第一个标准利用模型行为,优先选择LLM一贯出错的样本,这基于这样的直觉:这些样本可以暴露并纠正系统性的模型弱点。第二个标准基于查询程序的语义相似性来选择示例,使用k-最近邻检索来识别相关上下文。我们使用开源LLMs和涵盖多种编程语言的数据集进行了广泛的评估。我们的结果表明,对于Python和JavaScript,仔细选择少样本示例可以导致漏洞检测性能的可测量改进。相比之下,对于C和C++程序,少样本示例选择的影响有限,这表明可能需要更强大但成本更高的方法,如重新训练或微调,以显著提高模型性能。

英文摘要:

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of coding tasks, including summarization, translation, completion, and code generation. Despite these advances, detecting code vulnerabilities remains a challenging problem for LLMs. In-context learning (ICL) has emerged as an effective mechanism for improving model performance by providing a small number of labeled examples within the prompt. Prior work has shown, however, that the effectiveness of ICL depends critically on how these few-shot examples are selected. In this paper, we study two intuitive criteria for selecting few-shot examples for ICL in the context of code vulnerability detection. The first criterion leverages model behavior by prioritizing samples on which the LLM consistently makes mistakes, motivated by the intuition that such samples can expose and correct systematic model weaknesses. The second criterion selects examples based on semantic similarity to the query program, using k-nearest-neighbor retrieval to identify relevant contexts. We conduct extensive evaluations using open-source LLMs and datasets spanning multiple programming languages. Our results show that for Python and JavaScript, careful selection of few-shot examples can lead to measurable performance improvements in vulnerability detection. In contrast, for C and C++ programs, few-shot example selection has limited impact, suggesting that more powerful but also more expensive approaches, such as re-training or fine-tuning, may be required to substantially improve model performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑