在微调之前进行评估：针对网络安全问答的小型语言模型诊断研究
Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
浏览论文内容
中文总结 AI 辅助
针对网络安全问答选择小型语言模型难的问题，提出FiT诊断框架，从三方面刻画模型。通过对五个模型在两种微调模式下研究发现微调利弊因模式而异，预微调FiT分数可预测变化，能筛选模型、避免不必要微调，支持安全部署。
中文摘要 AI 辅助
大语言模型（LLMs）越来越多地针对关键领域的问答（QA）进行微调，但在付出适应成本之前，选择合适的小型模型仍然困难。微调虽能改善领域对齐，但可能侵蚀先验知识、削弱指令遵循或增加幻觉，尤其在网络安全领域，数据稀缺且快速演变。我们提出FiT（微调前评估），一个面向任务的诊断框架，从词汇识别、参数知识和检索信息的上下文关联三个能力来刻画小型语言模型。通过FiT对五个70亿参数的开放权重模型在两种微调模式下进行实证研究。结果表明微调并非总是有益，不同模式有不同权衡，预微调的FiT分数能预测微调后变化方向。研究结果表明面向任务的诊断可筛选不合适模型，避免不必要微调，支持小型LLMs在网络安全QA管道中更安全地部署。
英文摘要
Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.
发表机构
- The University of Alabama（阿拉巴马大学）
- Meharry Medical College（梅哈里医学院）
- Mississippi State University（密西西比州立大学）
- The University of Texas at El Paso（德克萨斯大学埃尔帕索分校）
机构由 AI 辅助整理，请以论文原文为准。
