基于领域适应语言模型的威胁建模：实证评估与洞察
Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights
AI总结：
本文通过评估不同规模的领域适应语言模型，探讨了领域适应、模型规模、解码策略和提示技术对STRIDE威胁分类的影响，揭示了当前LLM在结构化威胁建模中的局限性。
摘要：
大型语言模型（LLMs）越来越多地被用于网络安全应用，如漏洞检测。在威胁建模领域，先前的工作主要在有限的提示设置下评估了多种通用大型语言模型。本研究通过系统评估不同规模的领域适应语言模型，扩展了结构化威胁建模的研究范围。我们使用了针对电信和网络安全领域适应的LLM和小型语言模型（SLMs）。对于结构化威胁建模，我们选择了广泛使用的STRIDE方法，并应用于5G安全领域。我们通过52种不同的配置（在8种不同语言模型上）进行综合实证评估，分析了领域适应、模型规模、解码策略（贪心 vs. 随机采样）和提示技术对STRIDE威胁分类的影响。我们的结果表明，领域适应模型并不总能超越其通用模型，解码策略显著影响模型行为和输出有效性。它们还显示，尽管较大模型通常能获得更高的性能，但这些增益既不一致也不足以实现可靠的威胁建模。这些发现突显了当前LLM在结构化威胁建模任务中的根本限制，并表明改进需要比增加训练数据或模型规模更多，需要融入更多任务特定的推理和更强的安全概念基础。我们还提出了关于无效输出的见解，并提出了专门针对STRIDE威胁建模的提示建议。
英文摘要：
Large Language Models(LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection. In the domain of threat modelling, prior work has primarily evaluated a number of general-purpose Large Language Models under limited prompting settings. In this study, we extend the research area of structured threat modelling by systematically evaluating domain-adapted language models of different sizes to their general counterparts. We use both LLMs and Small Language Models(SLMs) that were domain adapted to telecommunications and cybersecuirty. For the structured threat modelling, we selected the widely used STRIDE approach and the application area is 5G security. We present a comprehensive empirical evaluation using 52 different configurations (on 8 different language models) to analyze the impact of 1) domain adaptation, 2) model scale, 3) decoding strategies (greedy vs. stochastic sampling), and 4) prompting technique on STRIDE threat classification. Our results show that domain-adapted models do not consistently outperform their general-purpose counterparts, and decoding strategies significantly affect model behavior and output validity. They also show that while larger models generally achieve higher performance, these gains are neither consistent nor sufficient for reliable threat modelling. These findings highlight fundamental limitations of current LLMs for structured threat modelling tasks and suggest that improvements require more than additional training data or model scaling, motivating the need for incorporating more task-specific reasoning and stronger grounding in security concepts. We present insights on invalid outputs encountered and present suggestions for prompting tailored specifically for STRIDE threat modelling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
