arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.05797cs.CL
收藏

演示如何修改:使用XLNet改进词汇约束的句子生成

Show Me How To Revise: Improving Lexically Constrained Sentence Generation with XLNet

The University of Hong Kong(香港大学)

Xingwei He, Victor O. K. Li

摘要:

词汇约束的句子生成允许将诸如词汇约束的先验知识并入输出。该技术已应用于机器翻译和对话响应生成。以前的工作通常使用马尔可夫链蒙特卡罗(MCMC)抽样来生成词汇限制的句子,但他们随机确定要编辑的位置和要采取的行动,导致许多无效的改进。为了克服这一挑战,我们使用了一个分类器来指导基于MCMC的模型在哪里以及如何细化候选句子。首先,我们开发了两种方法来创建合成数据,在此基础上对预训练模型进行微调,以获得可靠的分类器。接下来,我们提出了一个两步的方法,“预测和修改”,约束句子生成。在预测步骤中,我们利用分类器来计算候选句子的学习先验。在修改步骤中,我们采取MCMC采样,通过在从学习到的先验知识中抽取的采样位置进行采样动作来修改候选句子。我们将我们提出的模型与两个任务的许多强基线进行了比较,即生成具有词汇约束的句子和文本填充。实验结果表明,我们提出的模型表现出更好的句子流畅性和多样性方面比以前的工作。我们的代码和预培训模型可在https://github.com/NLPCode/MCMCXLNet上获得。

英文摘要:

Lexically constrained sentence generation allows the incorporation of prior knowledge such as lexical constraints into the output. This technique has been applied to machine translation, and dialog response generation. Previous work usually used Markov Chain Monte Carlo (MCMC) sampling to generate lexically constrained sentences, but they randomly determined the position to be edited and the action to be taken, resulting in many invalid refinements. To overcome this challenge, we used a classifier to instruct the MCMC-based models where and how to refine the candidate sentences. First, we developed two methods to create synthetic data on which the pre-trained model is fine-tuned to obtain a reliable classifier. Next, we proposed a two-step approach, "Predict and Revise", for constrained sentence generation. During the predict step, we leveraged the classifier to compute the learned prior for the candidate sentence. During the revise step, we resorted to MCMC sampling to revise the candidate sentence by conducting a sampled action at a sampled position drawn from the learned prior. We compared our proposed models with many strong baselines on two tasks, generating sentences with lexical constraints and text infilling. Experimental results have demonstrated that our proposed model performs much better than the previous work in terms of sentence fluency and diversity. Our code and pre-trained models are available at https://github.com/NLPCode/MCMCXLNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑