GR-SAP: 生成性重放用于在微调过程中保持安全对齐
GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning
AI总结:
GR-SAP通过生成领域特定对齐数据来保持微调过程中的安全对齐,有效缓解了微调导致的安全退化问题。
摘要:
近期研究表明,即使是由看似非对抗性的微调引起的大型语言模型(LLMs)的安全对齐性也可能轻易被破坏。为了在微调过程中保持安全对齐性,一种广泛使用的策略是通过混合原始对齐数据来联合优化安全性和任务目标,但这种原始对齐数据甚至对于开源权重的LLMs来说也是难以获取的。受持续学习中生成性重放的启发,我们提出了生成性安全对齐重放(GR-SAP),这是一个统一的框架,该框架从LLMs中合成领域特定的对齐数据,并在下游适应过程中整合这些数据以保持安全对齐性。理论和实证分析证明,这种合成数据可以作为原始对齐数据的可靠代理。在各种模型和下游任务上的实验表明,GR-SAP显著减轻了微调引起的安全退化,同时保持了可比的下游性能。我们的代码可在https://github.com/chili-lab/gr-sap上获得。
英文摘要:
Recent studies show that the safety alignment of large language models (LLMs) can be easily compromised even by seemingly non-adversarial fine-tuning. To preserve safety alignment during fine-tuning, a widely used strategy is to jointly optimize safety and task objectives by mixing in the original alignment data, which is typically inaccessible even for open-weight LLMs. Inspired by generative replay in continual learning, we propose Generative Replay for Safety Alignment Preservation (GR-SAP), a unified framework that synthesizes domain-specific alignment data from LLMs and integrate them during downstream adaption to preserve safety alignment. Theoretical and empirical analyses demonstrate this synthetic data serves as a reliable proxy for the original alignment data. Experiments across various models and downstream tasks show that GR-SAP substantially mitigates fine-tuning-induced safety degradation while maintaining comparable downstream performance. Our code is available at https://github.com/chili-lab/gr-sap.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。