arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.14989cs.CL
收藏

语言模型中的结构持久性:作为抽象语言表示窗口的启动

Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations

University of Aberdeen(阿伯丁大学);University of Amsterdam(阿姆斯特丹大学)

Arabella Sinclair, Jaap Jumelet, Willem Zuidema, Raquel Fernández

摘要:

我们研究现代神经语言模型在多大程度上容易受到结构启动的影响,即句子结构使后续句子中的相同结构更有可能的现象。我们探讨了如何启动可以用来研究这些模型的潜力,学习抽象的结构信息,这是一个先决条件,需要自然语言理解技能的任务表现良好。我们引入了一种新的度量和释放Prime-LM,一个大型语料库,我们控制各种语言因素与启动强度。我们发现,Transformer模型确实显示出结构启动的证据,但他们学到的概括在一定程度上受到语义信息的调制。我们的实验还表明,由模型获得的表示可能不仅编码抽象的顺序结构,但涉及一定程度的层次句法信息。更一般地说,我们的研究表明,启动范式是一个有用的,额外的工具,获得洞察能力的语言模型,并打开了大门,未来启动为基础的调查,探讨模型的内部状态。

英文摘要:

We investigate the extent to which modern, neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors which interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalisations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model's internal states.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑