arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.01044cs.CL
收藏

用人工语言检验神经语言模型的归纳偏差

Examining the Inductive Bias of Neural Language Models with Artificial Languages

University of Cambridge(剑桥大学);ETH Zürich(苏黎世联邦理工学院)

Jennifer C. White, Ryan Cotterell

摘要:

由于语言模型用于对各种语言进行建模,因此很自然地会问用于任务的神经架构是否对特定类型的语言建模有归纳偏见。这些偏差的调查已被证明是复杂的,由于在实验设置中出现的许多变量。语言在许多类型学维度上都有差异,很难挑出一两种来进行研究,而不把其他语言作为混杂因素。我们提出了一种新的方法来调查的归纳偏见的语言模型使用人工语言。这些语言的构建是为了让我们能够创建跨语言的平行语料库,这些语言之间的差异仅在于所研究的类型学特征,如词序。然后我们使用它们来训练和测试语言模型。这构成了一个完全受控的因果框架,并展示了语法工程如何成为分析神经模型的有用工具。使用这种方法,我们发现,常用的神经架构表现出不同的归纳偏见:LSTM显示关于单词排序的偏好很小,而Transformers显示出明显的偏好,一些排序超过其他。此外,我们发现LSTM的归纳偏差和Transformer的归纳偏差似乎都没有反映出我们在经过验证的自然语言中看到的任何趋势。

英文摘要:

Since language models are used to model a wide variety of languages, it is natural to ask whether the neural architectures used for the task have inductive biases towards modeling particular types of languages. Investigation of these biases has proved complicated due to the many variables that appear in the experimental setup. Languages vary in many typological dimensions, and it is difficult to single out one or two to investigate without the others acting as confounders. We propose a novel method for investigating the inductive biases of language models using artificial languages. These languages are constructed to allow us to create parallel corpora across languages that differ only in the typological feature being investigated, such as word order. We then use them to train and test language models. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models. Using this method, we find that commonly used neural architectures exhibit different inductive biases: LSTMs display little preference with respect to word ordering, while transformers display a clear preference for some orderings over others. Further, we find that neither the inductive bias of the LSTM nor that of the transformer appears to reflect any tendencies that we see in attested natural languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑