让模型回应:从提示依赖的角度解读语言模型去毒化
Let the Models Respond: Interpreting Language Model Detoxification Through the Lens of Prompt Dependence
AI总结:
本研究通过特征归因方法,对比反叙事微调与强化学习去毒化对语言模型提示依赖的影响,发现两者性能相似但机制不同。
摘要:
由于语言模型倾向于生成有毒或仇恨性的回应,研究人员开发了多种技术来使模型生成内容与用户偏好对齐。尽管这些方法在提升模型交互安全性方面效果显著,但它们对模型内部过程的影响仍知之甚少。在本工作中,我们将流行的去毒化方法应用于多个语言模型,并使用特征归因方法量化这些方法对所得模型的提示依赖性的影响。我们评估了反叙事微调的有效性,并将其与基于强化学习的去毒化方法进行比较,观察到尽管两种方法的去毒化性能相似,但它们在提示依赖方面存在差异。
英文摘要:
Due to language models' propensity to generate toxic or hateful responses, several techniques were developed to align model generations with users' preferences. Despite the effectiveness of such methods in improving the safety of model interactions, their impact on models' internal processes is still poorly understood. In this work, we apply popular detoxification approaches to several language models and quantify their impact on the resulting models' prompt dependence using feature attribution methods. We evaluate the effectiveness of counter-narrative fine-tuning and compare it with reinforcement learning-driven detoxification, observing differences in prompt reliance between the two methods despite their similar detoxification performances.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。