WPN:语言模型中基于N对对比学习的遗忘方法
WPN: An Unlearning Method Based on N-pair Contrastive Learning in Language Models
AI总结:
针对生成式语言模型预训练习得的有害知识易生成有害输出、现有遗忘方法损害模型性能的问题,提出基于N对对比学习的WPN方法,在高无害率下维持模型性能且泛化性、鲁棒性良好。
摘要:
生成式语言模型(LMs)具备诸多优势,但由于预训练阶段习得的有害知识,可能会生成不当或有害的输出。这类知识通常表现为不良对应关系,例如“有害提示”引发“有害输出”,本研究旨在通过遗忘技术缓解该问题。然而,现有基于梯度上升的遗忘方法会大幅损害语言模型的性能。为解决这一问题,我们提出了一种名为加权位置N对(Weighted Positional N-pair, WPN)学习的新方法,该方法在N对对比学习框架内利用位置加权均值池化。WPN旨在通过消除特定有害输出(例如将毒性回复替换为中性回复)来调整语言模型的输出分布,从而将模型行为从“有害提示-有害输出”转变为“有害提示-无害回复”。在OPT和GPT-NEO语言模型上的实验表明,WPN有效降低了有害回复的比例,无害率最高可达95.8%,同时在9个常用基准上保持稳定性能(平均性能下降不足2%)。此外,我们通过分布外测试集和对抗攻击场景下的评估,提供了实证证据,证明WPN在泛化性和鲁棒性方面具备削弱有害对应关系的能力。
英文摘要:
Generative language models (LMs) offer numerous advantages but may produce inappropriate or harmful outputs due to the harmful knowledge acquired during pre-training. This knowledge often manifests as undesirable correspondences, such as "harmful prompts" leading to "harmful outputs," which our research aims to mitigate through unlearning techniques.However, existing unlearning methods based on gradient ascent can significantly impair the performance of LMs. To address this issue, we propose a novel approach called Weighted Positional N-pair (WPN) Learning, which leverages position-weighted mean pooling within an n-pair contrastive learning framework. WPN is designed to modify the output distribution of LMs by eliminating specific harmful outputs (e.g., replacing toxic responses with neutral ones), thereby transforming the model's behavior from "harmful prompt-harmful output" to "harmful prompt-harmless response".Experiments on OPT and GPT-NEO LMs show that WPN effectively reduces the proportion of harmful responses, achieving a harmless rate of up to 95.8\% while maintaining stable performance on nine common benchmarks (with less than 2\% degradation on average). Moreover, we provide empirical evidence to demonstrate WPN's ability to weaken the harmful correspondences in terms of generalizability and robustness, as evaluated on out-of-distribution test sets and under adversarial attacks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。