arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1803.05071cs.CL
收藏

神经格语言模型

Neural Lattice Language Models

Language Technologies Institute(语言技术研究所);Carnegie Mellon University(卡内基梅隆大学)

Jacob Buckman, Graham Neubig

摘要:

在这项工作中,我们提出了一种新的语言建模范式,有能力在多个粒度执行预测和调节信息流:神经网格语言模型。这些模型构建了一个通过句子的可能路径的网格,并在这个网格上边缘化以计算序列概率或优化参数。这种方法使我们能够无缝地将语言直觉-包括多义词和多词词汇项的存在-纳入我们的语言模型。多语言建模任务的实验表明,利用多义嵌入的英语神经网格语言模型相对于单词级基线能够将困惑度提高9.95%,而处理多字符标记的中文模型相对于字符级基线能够将困惑度提高20.94%。

英文摘要:

In this work, we propose a new language modeling paradigm that has the ability to perform both prediction and moderation of information flow at multiple granularities: neural lattice language models. These models construct a lattice of possible paths through a sentence and marginalize across this lattice to calculate sequence probabilities or optimize parameters. This approach allows us to seamlessly incorporate linguistic intuitions - including polysemy and existence of multi-word lexical items - into our language model. Experiments on multiple language modeling tasks show that English neural lattice language models that utilize polysemous embeddings are able to improve perplexity by 9.95% relative to a word-level baseline, and that a Chinese model that handles multi-character tokens is able to improve perplexity by 20.94% relative to a character-level baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑