arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2005.00165cs.CL
收藏

循环神经网络语言模型始终学习英语类关系从句附加

Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

Cornell University(康奈尔大学)

Forrest Davis, Marten van Schijndel

摘要:

评估语言模型的标准方法分析模型如何将概率分配给有效与无效的句法结构(即,语法句子比不语法句子更有可能)。我们的工作使用歧义关系从句附件扩展这样的评估的情况下,多个同时有效的解释,其中明显的语法差异是缺席。我们比较了英语和西班牙语的模型性能,结果表明RNN LM中的非语言偏见与英语而不是西班牙语的句法结构有利地重叠。因此,英语模型可能会获得类似人类的句法偏好,而在西班牙语上训练的模型则无法获得类似人类的偏好。最后,我们将这些结果与理解(即典型的语言模型用例)和生成(生成语言模型的训练数据)之间的关系联系起来,这表明训练信号中根本不存在必要的语言偏见。

英文摘要:

A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑