RNN可以学习递归嵌套的主语-动词协议吗？
Can RNNs learn Recursive Nested Subject-Verb Agreements?
摘要：
当代语言学的一个基本原则是，语言处理需要能够提取递归嵌套的树结构。然而，目前还不清楚这种代码是否以及如何在神经回路中实现。循环神经网络（RNN）的最新进展，在某些语言任务中实现了接近人类的性能，为解决这些问题提供了一个令人信服的模型。在这里，我们提出了一个新的框架来研究RNN中的递归处理，使用主语-动词一致性作为对神经网络表示的探索。我们在一个简化的概率上下文无关文法上训练了六种不同类型的RNN，该文法旨在独立地操纵句子的长度及其句法树的深度。所有RNN都比训练期间看到的更长时间地推广到主语-动词依赖关系。然而，没有一个系统地推广到更深的树结构，即使是那些对学习嵌套树有结构偏见的结构（即，stack-RNNs）。此外，我们的分析揭示了基于LSTM的模型的泛化模式中的首因效应和近因效应，表明这些模型往往在中心嵌入式树结构的外部和内部表现良好，但在中间层表现不佳。最后，通过对嵌套树结构句子处理过程中模型内部状态的分析，我们发现了语法一致性信息（如语法数）的复杂编码，其中多词名词的所有信息都由一个单位携带。总之，这些结果表明神经网络可以提取有界嵌套树结构，而无需学习系统的递归规则。
英文摘要：
One of the fundamental principles of contemporary linguistics states that language processing requires the ability to extract recursively nested tree structures. However, it remains unclear whether and how this code could be implemented in neural circuits. Recent advances in Recurrent Neural Networks (RNNs), which achieve near-human performance in some language tasks, provide a compelling model to address such questions. Here, we present a new framework to study recursive processing in RNNs, using subject-verb agreement as a probe into the representations of the neural network. We trained six distinct types of RNNs on a simplified probabilistic context-free grammar designed to independently manipulate the length of a sentence and the depth of its syntactic tree. All RNNs generalized to subject-verb dependencies longer than those seen during training. However, none systematically generalized to deeper tree structures, even those with a structural bias towards learning nested tree (i.e., stack-RNNs). In addition, our analyses revealed primacy and recency effects in the generalization patterns of LSTM-based models, showing that these models tend to perform well on the outer- and innermost parts of a center-embedded tree structure, but poorly on its middle levels. Finally, probing the internal states of the model during the processing of sentences with nested tree structures, we found a complex encoding of grammatical agreement information (e.g. grammatical number), in which all the information for multiple words nouns was carried by a single unit. Taken together, these results indicate how neural networks may extract bounded nested tree structures, without learning a systematic recursive rule.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
