Safe to Serve：对齐指令微调模型以实现安全性与有用性
Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
AI总结：
本文研究如何在不牺牲有用性的前提下提升指令微调LLM的安全性，发现引入安全指令并采用DPO可将安全回复率由40%提升至90%以上。
摘要：
大型语言模型（LLM）在复杂推理和文本生成方面展现出卓越能力。然而，当被给予有问题的输入提示时，这些模型可能会无意中生成不安全或带有偏见的回复，从而给实际部署带来重大的伦理和现实担忧。本研究应对开发既能生成有用内容又能生成无害内容的语言模型这一关键挑战，在模型性能与安全性之间把握微妙平衡。我们证明，在对预训练模型进行指令微调期间纳入与安全相关的指令，可显著减少对不安全提示的有毒回复，同时不会损害在有用性数据集上的性能。我们发现 Direct Preference Optimization（DPO，直接偏好优化）尤其有效，它通过同时利用所选回复和被拒回复进行学习，表现优于 SIT 和 RAFT。我们的方法在各类有害性基准测试中将安全回复比例从40%提高到90%以上。此外，我们讨论了一个严格的评估框架，该框架涵盖用于安全性和有用性任务的专门指标与多样化数据集，可确保对模型能力进行全面评估。
英文摘要：
Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning and text generation. However, these models can inadvertently generate unsafe or biased responses when prompted with problematic inputs, raising significant ethical and practical concerns for real-world deployment. This research addresses the critical challenge of developing language models that generate both helpful and harmless content, navigating the delicate balance between model performance and safety. We demonstrate that incorporating safety-related instructions during the instruction-tuning of pre-trained models significantly reduces toxic responses to unsafe prompts without compromising performance on helpfulness datasets. We found Direct Preference Optimization (DPO) to be particularly effective, outperforming both SIT and RAFT by leveraging both chosen and rejected responses for learning. Our approach increased safe responses from 40$\%$ to over 90$\%$ across various harmfulness benchmarks. In addition, we discuss a rigorous evaluation framework encompassing specialized metrics and diverse datasets for safety and helpfulness tasks ensuring a comprehensive assessment of the model's capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
