从SFT到RL：解开基于LLM的漏洞检测后训练流程的谜团
From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection
AI总结：
本文研究了基于LLM的漏洞检测后训练流程，发现基于GRPO的在线RL优于SFT和离线偏好优化方法，提出了数据筛选、阶段交互、奖励机制和评估协议的改进策略。
摘要：
将LLM整合到漏洞检测（VD）中使该领域转向更可解释和上下文感知的分析。虽然后训练技术在一般编码任务中表现出色，但其在VD中的系统应用仍待探索。本文首次全面研究了基于LLM的VD后训练流程，证明在线RL与GRPO一致地优于SFT、离线偏好优化方法和专门的VD LLMs。我们的研究还揭示了VD特定的后训练指南和见解：（1）在数据筛选方面，与先前VD工作的广泛使用理性化监督不同，基于拒绝采样的SFT更有效，因为理性化可能引入幻觉；在RL训练中，漏洞固有的偏斜难度分布导致难度感知数据筛选大幅减少数据覆盖，造成非可忽略的性能损失，并削弱了课程学习，而基于配对的数据调度可以部分缓解这一问题。（2）在阶段交互方面，不同于偏好优化通常应用于轻度训练的SFT模型，增加SFT轮次一致地有助于VD任务中的离线偏好优化；然而，过度的SFT会抑制在线RL中的自我探索，限制其收益。（3）在奖励机制方面，简单地将漏洞分类正确性视为奖励信号会导致奖励黑客，而细粒度的根本原因判断提供更可靠的信用分配；基于规范的奖励进一步提高效率，但需要额外的设计和生成努力。（4）在评估协议方面，基于根本原因分析的LLM-as-a-Judge提供更稳健的替代方案，尽管在不同判官模型之间存在变异性。
英文摘要：
The integration of LLMs into vulnerability detection (VD) has shifted the field toward more interpretable and context-aware analysis. While post-training techniques have shown promise in general coding tasks, their systematic application to VD remains underexplored. In this paper, we present the first comprehensive investigation into the post-training pipeline for LLM-based VD, demonstrating that on-policy RL with GRPO consistently outperforms SFT, off-policy preference optimization methods, and specialized VD LLMs. Our study further reveals VD-specific post-training guidelines and insights beyond common practices: (1) For data curation, contrary to the widespread use of rationalization-based supervision in prior VD work, SFT based on rejection sampling proves more effective, as rationalization can introduce hallucinations; in RL training, the inherently skewed difficulty distribution of vulnerabilities leads difficulty-aware data filtering to drastically reduce data coverage, causing non-negligible performance loss, and undermines curriculum learning, while pair-based data scheduling can partially mitigate this. (2) For stage interactions, unlike preference optimization typically applied to lightly trained SFT models, increasing SFT epochs consistently benefits off-policy preference optimization in VD tasks; however, excessive SFT suppresses self-exploration in on-policy RL, limiting its gains. (3) For reward mechanisms, naively treating vulnerability classification correctness as reward signals leads to reward hacking, whereas fine-grained root-cause judgments provide more reliable credit assignment; specification-based rewards further improve efficiency at the cost of additional design and generation effort. (4) For evaluation protocols, LLM-as-a-Judge based on root-cause analysis offers a more robust alternative, albeit with variability across judge models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
