神经网络学习自然语言背后的统计规律吗？
Do Neural Nets Learn Statistical Laws behind Natural Language?
摘要：
深度学习在自然语言处理中的表现非常出色，但由于深度学习固有的复杂性，这种成功的原因仍然不清楚。本文提供了经验证据的有效性和语言工程的神经网络的局限性。准确地说，我们证明了基于长短期记忆（LSTM）的神经语言模型有效地再现了Zipf定律和Heaps定律，这是自然语言的两个代表性统计特性。我们讨论了质量的再现性和出现的齐普夫定律和堆的法律培训的进展。我们还指出，神经语言模型在再现自然语言的另一个统计特性--长程相关性方面存在局限性。这种理解可以为改进神经网络的体系结构提供方向。
英文摘要：
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law and Heaps' law as training progresses. We also point out that the neural language model has a limitation in reproducing long-range correlation, another statistical property of natural language. This understanding could provide a direction for improving the architectures of neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
