大型语言模型(LLMs)在漏洞检测方面展示了显著的 proficiency。然而,存在一个关键的可靠性缺口:模型经常基于幻觉逻辑或表面模式得出正确的检测结论,这些模式偏离实际的根本原因。这种不一致仍然 largely 被掩盖,因为当前的基准测试主要优先考虑粗粒度分类指标,缺乏评估底层推理过程所需的细粒度真实数据。为弥合这一缺口,我们首先构建了一个包含两个数据集的基准测试:(1)具有专家整理因果推理的真实世界漏洞,以及(2)语义等价的代码扰动用于评估推理鲁棒性。我们的大规模实证研究揭示,即使是最先进的模型在语义代码理解中也难以维持逻辑一致性,表现出12种系统性失败模式。为解决这些限制,我们提出DAGVul,一种将漏洞推理建模为有向无环图(DAG)生成任务的新框架。与线性链式思维(CoT)不同,我们的方法明确地将因果依赖性映射到强制结构一致性。通过进一步引入可验证奖励的强化学习(RLVR),我们对齐模型推理轨迹与程序内在逻辑。实验结果表明,我们的框架在所有基线上将推理F1分数平均提高了18.9%。令人印象深刻的是,我们的8B参数实现不仅优于现有同等规模的模型,还超过了专门设计的大规模推理模型,包括Qwen3-30B-Reasoning和GPT-OSS-20B-High。它甚至与最先进的模型如Claude-Sonnet-4.5(75.47% vs. 76.11%)竞争,建立了在模型规模上的新效率。
英文摘要:
Large Language Models (LLMs) have demonstrated remarkable proficiency in vulnerability detection. However, a critical reliability gap persists: models frequently yield correct detection verdicts based on hallucinated logic or superficial patterns that deviate from the actual root cause. This misalignment remains largely obscured because contemporary benchmarks predominantly prioritize coarse-grained classification metrics, lacking the granular ground truth required to evaluate the underlying reasoning process. To bridge this gap, we first construct a benchmark consisting of two datasets: (1) real-world vulnerabilities with expert-curated causal reasoning as ground truth, and (2) semantically equivalent code perturbations for assessing reasoning robustness. Our large-scale empirical study reveals that even state-of-the-art models struggle to maintain logical consistency during semantic code comprehension, exhibiting 12 systematic failure patterns. Addressing these limitations, we propose DAGVul, a novel framework that models vulnerability reasoning as a Directed Acyclic Graph (DAG) generation task. Unlike linear chain-of-thought (CoT), our approach explicitly maps causal dependencies to enforce structural consistency. By further introducing Reinforcement Learning with Verifiable Rewards (RLVR), we align model reasoning trace with program-intrinsic logic. Experimental results demonstrate that our framework improves the reasoning F1-score by an average of 18.9% over all the baselines. Remarkably, our 8B-parameter implementation not only outperforms existing models of comparable scale but also surpasses specialized large-scale reasoning models, including Qwen3-30B-Reasoning and GPT-OSS-20B-High. It is even competitive with state-of-the-art models like Claude-Sonnet-4.5 (75.47% vs. 76.11%), establishing new efficiency in vulnerability reasoning across model scales.
Enhancing Vulnerability Reports with Automated and Augmented Description Summarization
Department of Computer Science at the University of Central Florida(佛罗里达中央大学计算机科学系);Department of Computer Science at Northeastern Illinois University(北伊利诺伊大学计算机科学系);Department of Computer Science at Qatar University(卡塔尔大学计算机科学系);Ewha Womans University(成均馆大学)
VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models
Institute of BigData, Fudan University(复旦大学大数据研究院);School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Computer Science, Huazhong University of Science and Technology(华中科技大学计算机科学学院);Purple Mountain Laboratories(紫金山实验室);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院)
LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning
Nanyang Technological University, Singapore(南洋理工大学);The Hong Kong University of Science(香港科学与技术大学);MetaTrust Labs(MetaTrust实验室);Singapore Management University, Singapore(新加坡管理学院);University of Oregon, Eugene, OR, USA(俄勒冈大学)
Evaluating Large Language Models for Line-Level Vulnerability Localization
School of Software, Beihang University(北京航空航天大学软件学院);Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);Singapore Management University(新加坡管理大学)
Evaluating LLaMA 3.2 for Software Vulnerability Detection
Research Group on Intelligent Engineering and Computing for Advanced Innovation and Development (GECAD), School of Engineering of the Polytechnic of Porto(波尔图理工学院工程学院);INESC TEC/Polytechnic of Porto - School of Engineering(INESC TEC/波尔图理工学院工程学院)
Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning
School of Computer Science, University of Nottingham, Nottingham, United Kingdom(诺丁汉大学计算机科学学院);School of Informatics and Computing, Northern Arizona University, Flagstaff, United States of America(北亚利桑那大学信息学院);School of Computer Science, University of Nottingham, Ningbo, China(诺丁汉大学宁波校区)
PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)