arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11285cs.CRcs.CL
收藏

面向大语言模型的自蒸馏与跨模型蒸馏:用于拒绝模式对齐的有效方法

Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment

Jie Li, Yi Liu, Chongyang Liu, Xiaoning Ren, Ling Shi, Weisong Sun, Yinxing Xue

AI总结:

本文实证分析九种 LLM 的拒绝模式,提出自蒸馏与跨模型蒸馏对齐方法,显著提升拒绝率并减少不安全内容,跨模型蒸馏拒绝率接近 Claude3 的 94.51%。

摘要:

诸如 OpenAI 的 GPT 系列、Anthropic 的 Claude 和 Meta 的 LLaMa 等大语言模型(LLM)已在文本生成方面展现出卓越能力。然而,它们易受有毒提示影响,这带来了重大安全挑战。本文研究包括监督微调(SFT)和基于人类反馈的强化学习(RLHF)在内的对齐技术,以缓解这些风险。我们对九种 LLM 的拒绝模式开展实证研究,结果表明,具有统一拒绝模式的模型(如 Claude3)表现出更高的安全性。基于这些发现,我们提出自蒸馏和跨模型蒸馏方法以增强 LLM 安全性。结果显示,这些方法显著提高了拒绝率并减少了不安全内容,其中跨模型蒸馏实现了接近 Claude3 的 94.51% 的拒绝率。这些发现强调了基于蒸馏的对齐在保护 LLM 免受有毒提示攻击方面的潜力。

英文摘要:

Large Language Models (LLMs) like OpenAI's GPT series, Anthropic's Claude, and Meta's LLaMa have shown remarkable capabilities in text generation. However, their susceptibility to toxic prompts presents significant security challenges. This paper investigates alignment techniques, including Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), to mitigate these risks. We conduct an empirical study on refusal patterns across nine LLMs, revealing that models with uniform refusal patterns, such as Claude3, exhibit higher security. Based on these findings, we propose self-distilling and cross-model distilling methods to enhance LLM security. Our results show that these methods significantly improve refusal rates and reduce unsafe content, with cross-model distilling achieving refusal rates close to Claude3's 94.51%. These findings underscore the potential of distillation-based alignment in securing LLMs against toxic prompts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑