arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.06434cs.CLcs.AI
收藏

使用Transformers的主题语言生成

Topical Language Generation using Transformers

University of Denver(丹佛大学)

Rohola Zandie, Mohammad H. Mahoor

摘要:

大规模的基于transformer的语言模型(LM)在开放文本生成方面表现出令人印象深刻的能力。然而,控制生成的文本的属性,如主题,风格和情感是具有挑战性的,通常需要对模型架构进行重大更改,或者在新的监督数据上重新训练和微调模型。提出了一种将预先训练好的LM与主题建模信息相结合的主题语言生成方法。我们投的问题,使用贝叶斯概率公式与主题概率作为先验,LM概率的可能性,和主题语言生成概率作为后验。在学习模型的过程中,我们从用户提供的文档的自然结构中推导出主题概率分布。此外,我们通过引入新的参数和函数来影响生成的文本中呈现的主题特征的数量,从而扩展了我们的模型。这个特性将允许我们轻松地控制生成文本的主题属性。我们的实验结果表明,我们的模型优于国家的最先进的结果一致性,多样性和流畅性,同时在解码速度更快。

英文摘要:

Large-scale transformer-based language models (LMs) demonstrate impressive capabilities in open text generation. However, controlling the generated text's properties such as the topic, style, and sentiment is challenging and often requires significant changes to the model architecture or retraining and fine-tuning the model on new supervised data. This paper presents a novel approach for Topical Language Generation (TLG) by combining a pre-trained LM with topic modeling information. We cast the problem using Bayesian probability formulation with topic probabilities as a prior, LM probabilities as the likelihood, and topical language generation probability as the posterior. In learning the model, we derive the topic probability distribution from the user-provided document's natural structure. Furthermore, we extend our model by introducing new parameters and functions to influence the quantity of the topical features presented in the generated text. This feature would allow us to easily control the topical properties of the generated text. Our experimental results demonstrate that our model outperforms the state-of-the-art results on coherency, diversity, and fluency while being faster in decoding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑