投毒对 LLM 对齐真的是一种威胁吗？也许比你想象的更严重
Is poisoning a real threat to LLM alignment? Maybe more so than you think
AI总结：
本文系统评估 DPO 在后门与非后门偏好投毒下的安全风险，证明仅 0.5% 投毒数据即可诱发有害行为，显著低于 PPO 所需的 4%。
摘要：
基于人类反馈的强化学习（RLHF）的最新进展已显著影响大型语言模型（LLM）的对齐。近端策略优化（PPO）等强化学习算法的敏感性催生了关于直接策略优化（DPO）的一系列新工作，DPO 在监督学习框架中处理 RLHF。这些 RLHF 方法日益增长的实际应用，使得有必要分析其脆弱性。在本工作中，我们研究 DPO 在不同场景下面对投毒攻击的脆弱性，并首次比较偏好投毒的有效性。我们在多种语言模型（即 LLama 7B、Mistral 7B 和 Gemma 7B）上，全面分析 DPO 在不同攻击类型（即后门攻击与非后门攻击）以及不同投毒方法下的脆弱性。我们发现，与基于 PPO 的方法在面对后门攻击时至少需要 4% 的数据被投毒才能诱发有害行为不同，我们更直接地利用 DPO 的真实脆弱性，因此仅需投毒多达 0.5% 的数据即可投毒该模型。我们进一步研究这种脆弱性背后的潜在原因，以及该脆弱性在后门攻击与非后门攻击之间的转化程度。
英文摘要：
Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new line work on Direct Policy Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning, a first of its kind. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, i.e., LLama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4\% of the data to be poisoned to elicit harmful behavior, we exploit the true vulnerabilities of DPO more simply so we can poison the model with only as much as 0.5\% of the data. We further investigate the potential reasons behind the vulnerability and how well this vulnerability translates into backdoor vs non-backdoor attacks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
