arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.15335cs.CL
收藏

改变Transformers的思想以实现主题可控的语言生成

Changing the Mind of Transformers for Topically-Controllable Language Generation

University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, Andrew McCallum

摘要:

基于transformer的大型语言模型可以通过建议迄今为止所写文本的合理延续来帮助人类作者。然而,当前的交互式写作助手不允许作者在期望的主题方向上引导文本生成。为了解决这个问题,我们设计了一个框架,显示多个候选即将到来的主题,其中用户可以选择一个子集来指导生成。我们的框架由两部分组成:(1)一种方法,通过预测可能的延续中的词簇中心来产生一组候选主题,以及(2)一个文本生成模型,其输出符合所选主题。这两个组件的训练都是自我监督的,只使用未标记的文本。我们的实验表明,我们的主题选项比标准的聚类方法更好,我们的框架经常生成与所选主题相关的流畅句子,由自动化指标和众包工人判断。

英文摘要:

Large Transformer-based language models can aid human authors by suggesting plausible continuations of text written so far. However, current interactive writing assistants do not allow authors to guide text generation in desired topical directions. To address this limitation, we design a framework that displays multiple candidate upcoming topics, of which a user can select a subset to guide the generation. Our framework consists of two components: (1) a method that produces a set of candidate topics by predicting the centers of word clusters in the possible continuations, and (2) a text generation model whose output adheres to the chosen topics. The training of both components is self-supervised, using only unlabeled text. Our experiments demonstrate that our topic options are better than those of standard clustering approaches, and our framework often generates fluent sentences related to the chosen topics, as judged by automated metrics and crowdsourced workers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑