通过基于表示擦除的偏好优化来净化大语言模型
Detoxifying LLMs via Representation Erasure-Based Preference Optimization
AI总结：
本文提出REPO方法，通过基于表示擦除的偏好优化，有效净化大语言模型，提升其对抗鲁棒性。
摘要：
大规模语言模型（LLMs）在基于网络级数据进行训练时可能会生成有毒输出，这引发了安全部署的担忧。以往的防御措施基于DPO、NPO等算法的应用，减少了有害延续的可能性，但并不牢固：它们容易受到对抗性提示的攻击，并且容易被基于微调的再学习攻击所推翻。事实上，研究显示这些对模型的编辑是表面的：线性探测揭示有害
英文摘要：
Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
