发表机构
University of Central Florida（中佛罗里达大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出失败驱动的提示词精炼（FDPR）方法，通过分析模型在漏洞分析中的反复失败模式来系统改进提示词，实验证明该方法提升了可靠性并产生可复用设计原则。
AI 中文摘要
大型语言模型已成为软件漏洞分析的有前景工具，但其有效性在很大程度上依赖于提示词设计。现有研究主要使用聚合性能指标来比较提示词策略，对模型为何失败或如何系统地改进提示词提供的见解有限。我们提出了一种失败驱动的提示词精炼（FDPR）方法，该方法通过分析反复出现的模型失败来指导基于证据的提示词改进。利用易受攻击的Java应用程序（DVJA），我们识别了反复出现的失败模式，包括误报、漏报、无依据推理和CWE错误分类，并将这些模式转化为有针对性的提示词改进。随后，我们在Juliet测试套件上评估了由此产生的提示词，并进行了跨模型验证以评估其泛化能力。结果表明，失败驱动的精炼提高了基于LLM的漏洞分析的可靠性，同时产生了可复用的提示词设计原则。更广泛地说，这项工作表明，反复出现的模型失败为提示词工程提供了原则性基础，使得能够系统地开发更可靠的基于LLM的漏洞分析系统。
英文摘要
Large Language Models have emerged as promising tools for software vulnerability analysis, but their effectiveness depends heavily on prompt design. Existing research primarily compares prompting strategies using aggregate performance metrics, providing limited insight into why models fail or how prompts can be improved systematically. We propose Failure-Driven Prompt Refinement (FDPR), a methodology that analyzes recurring model failures to guide evidence-based prompt refinement. Using the Damn Vulnerable Java Application (DVJA), we identify recurring failure modes, including false positives, false negatives, unsupported reasoning, and CWE misclassification, and translate them into targeted prompt refinements. We then evaluate the resulting prompt on the Juliet Test Suite and perform cross-model validation to assess generalizability. The results show that failure-driven refinement improves the reliability of LLM-based vulnerability analysis while yielding reusable prompt design principles. More broadly, this work demonstrates that recurring model failures provide a principled foundation for prompt engineering, enabling the systematic development of more reliable LLM-based vulnerability analysis systems.
CommentsAccepted at CSoNet 2026. 15 pages, 6 figures/tables combined
