arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.10256cs.CL
收藏

基于依赖关系的混合语言模型

Dependency-based Mixture Language Models

Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室)

Zhixian Yang, Xiaojun Wan

摘要:

已经提出了各种模型来将句法结构的知识并入神经语言模型。然而,以前的工作在很大程度上依赖于特定语言模型的精细组件,通常是递归神经网络(RNN),这使得它们在实践中难以适应其他神经语言模型,如Transformer和GPT-2。本文介绍了基于依赖关系的混合语言模型。详细地说,我们首先训练神经语言模型与一个新的依赖建模目标,学习未来的依赖令牌给定的上下文的概率分布。然后,我们通过将先前的依赖建模概率分布与自注意混合来制定下一个令牌概率。大量的实验和人工评估表明,我们的方法可以轻松有效地应用于不同的神经语言模型,同时改善各种任务的神经文本生成。

英文摘要:

Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑