arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.08491cs.CLcs.LG
收藏

用于受控生成的对比困惑度:在大型语言模型去毒化中的应用

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models

SAP SE(SAP股份有限公司)

Tassilo Klein, Moin Nabi

AI总结:

提出基于原型的对比困惑度微调框架,通过对抗性释义构建困难负样本,在保持下游任务性能的同时显著减少大型语言模型的有毒内容生成。

摘要:

大型语言模型(LLMs)生成有毒内容仍是语言技术安全部署的关键挑战。我们提出一个新框架,通过基于原型的对比困惑度目标微调LLMs,实现隐式知识编辑与受控文本生成。该方法的核心是构建困难负样本——即通过对抗性释义生成的有毒输出,使其在语义上与非有毒对应物相似且模型概率相近。通过在这些具挑战性且真实的样本对上训练,该方法确保了稳健稳定的对比优化。去毒化领域的实验结果表明,该方法显著减少了有毒内容的生成,同时在常识推理和阅读理解等下游任务上保持了强劲的性能。研究结果凸显了利用困难负样本进行属性感知微调的有效性。

英文摘要:

The generation of toxic content by large language models (LLMs) remains a critical challenge for the safe deployment of language technology. We propose a novel framework for implicit knowledge editing and controlled text generation by fine-tuning LLMs with a prototype-based contrastive perplexity objective. Central to our method is the construction of hard negatives - toxic outputs that are generated through adversarial paraphrasing to be semantically similar and model probability to their non-toxic counterparts. By training on these challenging and realistic pairs, our approach ensures robust and stable contrastive optimization. Experimental results in the domain of detoxification demonstrate that our method significantly reduces toxic generation while maintaining strong performance on downstream tasks such as commonsense reasoning and reading comprehension. Our findings highlight the effectiveness of exploiting hard negatives for attribute-aware fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑