arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.14003cs.CLcs.AIcs.LG
收藏

基于死端分析的语言模型系统校正

Systematic Rectification of Language Models via Dead-end Analysis

Mila – Québec AI Institute(米拉-魁北克人工智能研究所);McGill University(麦吉尔大学);Microsoft Research(微软研究院)

Meng Cao, Mehdi Fatemi, Jackie Chi Kit Cheung, Samira Shabanian

摘要:

通过对抗性或其他正常提示,现有的大语言模型(LLM)可以被推到生成有毒话语。降低LLM生成不期望的话语的风险的一种方法是改变LLM的训练。这可能是非常严格的限制,由于苛刻的计算要求。其他方法依赖于基于规则或基于语义的标记消除,这是有限的,因为它们消除了未来的标记和完整话语的整体意义。在这里,我们把解毒的中心放在完成的话语最终被认为是有毒的可能性上。也就是说,在每一点上,我们都建议不要选择与从这一点完成的文本有毒的可能性成比例的标记。为此,我们从最近的强化学习(RL)文献中正式扩展了死胡同理论,以涵盖不确定的结果。我们的方法,称为整流,利用一个单独的,但明显较小的模型解毒,它可以适用于不同的LLM,只要他们共享相同的词汇。重要的是,我们的方法不需要访问LLM的内部表示,而只需要访问每个解码步骤的令牌概率分布。这是至关重要的,因为今天的许多LLM都托管在服务器上,只能通过API访问。当应用于各种LLM,包括GPT-3,我们的方法显着提高了生成的话语相比,基础LLM和其他技术的整体语言和解毒性能。

英文摘要:

With adversarial or otherwise normal prompts, existing large language models (LLM) can be pushed to generate toxic discourses. One way to reduce the risk of LLMs generating undesired discourses is to alter the training of the LLM. This can be very restrictive due to demanding computation requirements. Other methods rely on rule-based or prompt-based token elimination, which are limited as they dismiss future tokens and the overall meaning of the complete discourse. Here, we center detoxification on the probability that the finished discourse is ultimately considered toxic. That is, at each point, we advise against token selections proportional to how likely a finished text from this point will be toxic. To this end, we formally extend the dead-end theory from the recent reinforcement learning (RL) literature to also cover uncertain outcomes. Our approach, called rectification, utilizes a separate but significantly smaller model for detoxification, which can be applied to diverse LLMs as long as they share the same vocabulary. Importantly, our method does not require access to the internal representations of the LLM, but only the token probability distribution at each decoding step. This is crucial as many LLMs today are hosted in servers and only accessible through APIs. When applied to various LLMs, including GPT-3, our approach significantly improves the generated discourse compared to the base LLMs and other techniques in terms of both the overall language and detoxification performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑