神经语言模型中的层次表示：期望的抑制与恢复
Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations
摘要：
深度学习序列模型已经导致一系列自然语言处理任务的性能显著提高，但它们是否能够单独从线性输入中诱导适当的分层概括来表示自然语言仍然是一个悬而未决的问题。使用人工语言作为训练输入的工作表明，LSTM能够诱导表示上下文无关和某些轻度上下文敏感的语言所需的堆栈式数据结构--形式语言类，其在理论上对应于自然语言的层次结构。在这里，我们提出了一套实验，探讨在语言数据上训练的神经语言模型是否会诱导这些类似堆栈的数据结构，并在增量预测单词的同时部署它们。我们研究了两种自然语言现象：中心嵌入句和句法孤岛对填充语--空缺依赖的约束。为了正确预测这些结构中的单词，模型必须能够暂时抑制某些期望，然后稍后恢复这些期望，本质上是将这些期望推到堆栈上。我们的研究结果提供的证据表明，模型可以成功地抑制和恢复预期在许多情况下，但不能完全恢复他们以前的语法状态。
英文摘要：
Deep learning sequence models have led to a marked increase in performance for a range of Natural Language Processing tasks, but it remains an open question whether they are able to induce proper hierarchical generalizations for representing natural language from linear input alone. Work using artificial languages as training input has shown that LSTMs are capable of inducing the stack-like data structures required to represent context-free and certain mildly context-sensitive languages---formal language classes which correspond in theory to the hierarchical structures of natural language. Here we present a suite of experiments probing whether neural language models trained on linguistic data induce these stack-like data structures and deploy them while incrementally predicting words. We study two natural language phenomena: center embedding sentences and syntactic island constraints on the filler--gap dependency. In order to properly predict words in these structures, a model must be able to temporarily suppress certain expectations and then recover those expectations later, essentially pushing and popping these expectations on a stack. Our results provide evidence that models can successfully suppress and recover expectations in many cases, but do not fully recover their previous grammatical state.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
