arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06687cs.CR
收藏

评估和增强大型语言模型的漏洞推理能力

Evaluating and Enhancing the Vulnerability Reasoning Capabilities of Large Language Models

Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学)

Li Lu, Yanjie Zhao, Hongzhou Rao, Kechi Zhang, Haoyu Wang

AI总结:

DAGVul通过建模漏洞推理为有向无环图生成任务,提升大型语言模型在漏洞检测中的推理能力,实验显示其推理F1分数平均提高18.9%。

摘要:

大型语言模型(LLMs)在漏洞检测方面展示了显著的 proficiency。然而,存在一个关键的可靠性缺口:模型经常基于幻觉逻辑或表面模式得出正确的检测结论,这些模式偏离实际的根本原因。这种不一致仍然 largely 被掩盖,因为当前的基准测试主要优先考虑粗粒度分类指标,缺乏评估底层推理过程所需的细粒度真实数据。为弥合这一缺口,我们首先构建了一个包含两个数据集的基准测试:(1)具有专家整理因果推理的真实世界漏洞,以及(2)语义等价的代码扰动用于评估推理鲁棒性。我们的大规模实证研究揭示,即使是最先进的模型在语义代码理解中也难以维持逻辑一致性,表现出12种系统性失败模式。为解决这些限制,我们提出DAGVul,一种将漏洞推理建模为有向无环图(DAG)生成任务的新框架。与线性链式思维(CoT)不同,我们的方法明确地将因果依赖性映射到强制结构一致性。通过进一步引入可验证奖励的强化学习(RLVR),我们对齐模型推理轨迹与程序内在逻辑。实验结果表明,我们的框架在所有基线上将推理F1分数平均提高了18.9%。令人印象深刻的是,我们的8B参数实现不仅优于现有同等规模的模型,还超过了专门设计的大规模推理模型,包括Qwen3-30B-Reasoning和GPT-OSS-20B-High。它甚至与最先进的模型如Claude-Sonnet-4.5(75.47% vs. 76.11%)竞争,建立了在模型规模上的新效率。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable proficiency in vulnerability detection. However, a critical reliability gap persists: models frequently yield correct detection verdicts based on hallucinated logic or superficial patterns that deviate from the actual root cause. This misalignment remains largely obscured because contemporary benchmarks predominantly prioritize coarse-grained classification metrics, lacking the granular ground truth required to evaluate the underlying reasoning process. To bridge this gap, we first construct a benchmark consisting of two datasets: (1) real-world vulnerabilities with expert-curated causal reasoning as ground truth, and (2) semantically equivalent code perturbations for assessing reasoning robustness. Our large-scale empirical study reveals that even state-of-the-art models struggle to maintain logical consistency during semantic code comprehension, exhibiting 12 systematic failure patterns. Addressing these limitations, we propose DAGVul, a novel framework that models vulnerability reasoning as a Directed Acyclic Graph (DAG) generation task. Unlike linear chain-of-thought (CoT), our approach explicitly maps causal dependencies to enforce structural consistency. By further introducing Reinforcement Learning with Verifiable Rewards (RLVR), we align model reasoning trace with program-intrinsic logic. Experimental results demonstrate that our framework improves the reasoning F1-score by an average of 18.9% over all the baselines. Remarkably, our 8B-parameter implementation not only outperforms existing models of comparable scale but also surpasses specialized large-scale reasoning models, including Qwen3-30B-Reasoning and GPT-OSS-20B-High. It is even competitive with state-of-the-art models like Claude-Sonnet-4.5 (75.47% vs. 76.11%), establishing new efficiency in vulnerability reasoning across model scales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑