arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07754cs.CRcs.CL
收藏

对齐的艺术:细调方法如何有效地对齐和重新对齐训练后的LLM

The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

University of Electronic Science and Technology of China(电子科技大学);Flexera;CISPA Helmholtz Center for Information Security(CISPA亥姆霍兹信息安全中心);Nanyang Technological University(南洋理工大学)

Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes, Yang Zhang

AI总结:

研究探讨细调方法在对齐和重新对齐LLM中的效果,揭示攻击与防御间的机制不对称,发现ORPO在对齐方面最有效,DPO在重新对齐中表现优异但牺牲了模型实用性,同时发现模型特定的抗性及多轮对抗动态的残留效应。

摘要:

大型语言模型(LLM)的部署引发了重大伦理和安全关切。尽管对齐技术被采用以提高模型的安全性和可信度,但攻击者可以利用这些技术来破坏安全,导致所谓的“对齐问题”。发布恶意对齐的LLM可能放大危害。为此,在部署不可信的第三方LLM之前需要额外的安全对齐,称为“重新对齐”。本研究探讨细调方法在对齐、重新对齐及它们相互作用效果方面的有效性。通过在四个流行的安全对齐LLM上评估四种监督细调(SFT)和两种偏好细调(PFT)方法,揭示攻击与防御间的机制不对称。虽然Odds Ratio Preference Optimization(ORPO)在对齐方面最有效,Direct Preference Optimization(DPO)在重新对齐中表现优异,但牺牲了模型实用性。此外,我们识别出模型特定的抗性、多轮对抗动态的残留效应及其他值得注意的发现。这些发现突显了需要稳健的安全保障和定制的安全对齐策略以减轻LLM部署中的潜在风险。我们的代码可在https://github.com/zhangrui4041/The-Art-of-Mis-alignment获取。

英文摘要:

The deployment of large language models (LLMs) raises significant ethical and safety concerns. While LLM alignment techniques are adopted to improve model safety and trustworthiness, adversaries can exploit these techniques to undermine safety for malicious purposes, resulting in \emph{misalignment}. Misaligned LLMs may be published on open platforms to magnify harm. To address this, additional safety alignment, referred to as \emph{realignment}, is necessary before deploying untrusted third-party LLMs. This study explores the efficacy of fine-tuning methods in terms of misalignment, realignment, and the effects of their interplay. By evaluating four Supervised Fine-Tuning (SFT) and two Preference Fine-Tuning (PFT) methods across four popular safety-aligned LLMs, we reveal a mechanism asymmetry between attack and defense. While Odds Ratio Preference Optimization (ORPO) is most effective for misalignment, Direct Preference Optimization (DPO) excels in realignment, albeit at the expense of model utility. Additionally, we identify model-specific resistance, residual effects of multi-round adversarial dynamics, and other noteworthy findings. These findings highlight the need for robust safeguards and customized safety alignment strategies to mitigate potential risks in the deployment of LLMs. Our code is available at https://github.com/zhangrui4041/The-Art-of-Mis-alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑