自条件预训练语言模型
Self-conditioning pre-trained language models
AI总结：
该研究基于专家乘积公式，提出通过激活预训练Transformer语言模型中天然存在的少量专家单元实现自条件生成，无需微调即可引导文本、纠正性别偏见，且计算需求低，助力理解模型生成机制。
摘要：
本文旨在探究引导基于Transformer的预训练语言模型（TLMs）生成文本的机制。基于Hinton（1999）提出的专家乘积（Product of Experts）公式，我们描述了一种利用TLMs中天然存在的专家单元的生成机制。这类单元负责检测输入中的概念，并基于这些概念对文本生成进行条件约束。我们阐述了如何识别专家单元，以及如何在推理过程中激活它们，从而在生成的输出中诱导出任意期望的概念。我们发现，激活数量少得惊人的单元就足以引导文本生成（在一个拥有3.45亿参数的模型中仅需3个单元）。\n虽然本研究的目标是进一步了解TLMs的工作原理，但我们证明了该方法无需微调或使用额外参数即可有效实现条件约束，甚至适用于细粒度的同形异义概念。此外，我们表明该方法可用于纠正TLMs输出中存在的性别偏见，在所有评估的上下文中都实现了性别均衡。我们将该方法与FUDGE和PPLM-BoW进行了比较，结果表明我们的方法能够以更低的困惑度实现性别均衡。由于其简单性和极低的计算需求，该方法可供广泛的研究群体使用。本文的研究结果是理解TLMs生成机制的重要一步。
英文摘要：
In this paper we aim to investigate the mechanisms that guide text generation with pre-trained Transformer-based Language Models (TLMs). Grounded on the Product of Experts formulation by Hinton (1999), we describe a generative mechanism that exploits expert units which naturally exist in TLMs. Such units are responsible for detecting concepts in the input and conditioning text generation on such concepts. We describe how to identify expert units and how to activate them during inference in order to induce any desired concept in the generated output. We find that the activation of a surprisingly small amount of units is sufficient to steer text generation (as little as 3 units in a model with 345M parameters). While the objective of this work is to learn more about how TLMs work, we show that our method is effective for conditioning without fine-tuning or using extra parameters, even on fine-grained homograph concepts. Additionally, we show that our method can be used to correct gender bias present in the output of TLMs and achieves gender parity for all evaluated contexts. We compare our method with FUDGE and PPLM-BoW, and show that our approach is able to achieve gender parity at a lower perplexity. The proposed method is accessible to a wide audience thanks to its simplicity and minimal compute needs. The findings in this paper are a step forward in understanding the generative mechanisms of TLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
