arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1708.07252cs.CLcs.AI
收藏

神经网络语言建模研究

A Study on Neural Network Language Modeling

Dengliang Shi

摘要:

本文对神经网络语言建模进行了详尽的研究。不同的架构的基本神经网络语言模型的描述和检查。研究了对基本神经网络语言模型的几种不同改进,包括重要性采样、词类、缓存和双向递归神经网络(BiRNN),并对每种技术的优缺点进行了评价。然后从模型体系结构和知识表示两个方面探讨了神经网络语言建模的局限性。当按一定的顺序逐字处理时,词序列中的部分统计信息会丢失,而神经网络的权值矩阵和权值向量的更新机制严重限制了NNLM的有效增强。对于知识表示,神经网络语言模型所表示的知识是来自某个训练数据集的词序列的近似概率分布,而不是语言本身的知识或自然语言中的词序列所传达的信息。最后,对神经网络语言建模的进一步改进方向进行了讨论。

英文摘要:

An exhaustive study on neural network language modeling (NNLM) is performed in this paper. Different architectures of basic neural network language models are described and examined. A number of different improvements over basic neural network language models, including importance sampling, word classes, caching and bidirectional recurrent neural network (BiRNN), are studied separately, and the advantages and disadvantages of every technique are evaluated. Then, the limits of neural network language modeling are explored from the aspects of model architecture and knowledge representation. Part of the statistical information from a word sequence will loss when it is processed word by word in a certain order, and the mechanism of training neural network by updating weight matrixes and vectors imposes severe restrictions on any significant enhancement of NNLM. For knowledge representation, the knowledge represented by neural network language models is the approximate probabilistic distribution of word sequences from a certain training data set rather than the knowledge of a language itself or the information conveyed by word sequences in a natural language. Finally, some directions for improving neural network language modeling further is discussed.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑