用于安全思维链蒸馏的慢速微调与低熵掩码
Slow Tuning and Low-Entropy Masking for Safe Chain-of-Thought Distillation
AI总结：
本文揭示了思维链蒸馏对小型语言模型安全性的负面影响，提出SLowED方法，通过慢速微调缩小权重变化幅度并结合低熵掩码排除非必要学习目标，在不损害推理能力的前提下有效保持了模型安全性。
摘要：
先前的思维链（CoT）蒸馏方法主要侧重于通过利用强大的大型语言模型（LLMs，例如GPT-4）生成的高质量依据，来增强小型语言模型（SLMs）的推理能力。然而，很少有研究关注训练给SLM安全性带来的负面影响，本文揭示了这一问题。尽管已有关于安全对齐的工作通过微调语言模型或操纵模型权重来防御有害输入，但它们需要额外的计算或标注数据，且可能影响SLM的推理能力。在本文中，我们研究了如何在CoT蒸馏过程中保持SLM的安全性。具体而言，我们提出了一种安全的蒸馏方法——慢速微调与低熵掩码蒸馏（SLowED），包含两个模块：慢速微调和低熵掩码。慢速微调缩小模型权重变化的幅度，以在初始权重分布附近的邻近空间内优化模型权重。低熵掩码掩盖被视为不必要学习目标的低熵token，将其排除在微调之外。在三个SLM（Qwen2.5-1.5B、Llama-3.2-1B、BLOOM-1.1B）上跨推理基准（BBH、BB-Sub、ARC、AGIEval）和安全评估（AdvBench）的实验表明，与现有蒸馏方法相比，SLowED保留了SLM的安全性，并在可比程度上提升了其推理能力。此外，我们的消融研究证明了慢速微调和低熵掩码的有效性，前者在早期阶段维持模型的安全性，后者延长了安全训练的轮数。
英文摘要：
Previous chain-of-thought (CoT) distillation methods primarily focused on enhancing the reasoning capabilities of Small Language Models (SLMs) by utilizing high-quality rationales generated by powerful Large Language Models (LLMs, e.g., GPT-4). However, few works have noted the negative effects on SLM safety brought by the training, which are revealed in this study. Although there are works on safety alignment that fine-tune language models or manipulate model weights to defend against harmful inputs, they require extra computation or annotated data, and probably impact the reasoning ability of SLMs. In this paper, we investigate how to maintain the safety of SLMs during the CoT distillation process. Specifically, we propose a safe distillation method, Slow Tuning and Low-Entropy Masking Distillation (SLowED), containing two modules: Slow Tuning and Low-Entropy Masking. Slow Tuning scales down the magnitude of model weight changes to optimize the model weights in the neighboring space near the initial weight distribution. Low-Entropy Masking masks low-entropy tokens, which are regarded as unnecessary learning targets, to exclude them from fine-tuning. Experiments on three SLMs (Qwen2.5-1.5B, Llama-3.2-1B, BLOOM-1.1B) across reasoning benchmarks (BBH, BB-Sub, ARC, AGIEval) and safety evaluation (AdvBench) show that SLowED retains the safety of SLMs and comparably improves their reasoning capability compared to existing distillation methods. Furthermore, our ablation study presents the effectiveness of Slow Tuning and Low-Entropy Masking, with the former maintaining the model's safety in the early stage and the latter prolonging the safe training epochs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
