arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.12574cs.CLcs.LG
收藏

绝对位置嵌入的奇异情况

The Curious Case of Absolute Position Embeddings

McGill University(麦吉尔大学);Mila - Quebec AI(魁北克人工智能研究所米拉);Meta AI

Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, Adina Williams

摘要:

Transformer语言模型使用位置信息对词序的概念进行编码。最常见的是,这种位置信息由绝对位置嵌入(APE)表示,它是从预训练数据中学习的。然而,在自然语言中,重要的不是绝对位置,而是相对位置,并且APE可以捕获这种类型的信息的程度尚未被研究。在这项工作中,我们观察到用APE训练的模型过度依赖位置信息,以至于当遇到具有移动位置信息的句子时它们会崩溃。具体来说,当模型受到从非零位置开始的句子(不包括启动的影响)时,它们在一系列模型系列和模型大小的零到全拍摄任务上表现出明显的性能下降。我们的研究结果提出了问题的有效性APE模型的位置信息的相关性,并邀请进一步反省这些模型所采用的句子和词序处理策略。

英文摘要:

Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is represented by absolute position embeddings (APEs), that are learned from the pretraining data. However, in natural language, it is not absolute position that matters, but relative position, and the extent to which APEs can capture this type of information has not been investigated. In this work, we observe that models trained with APE over-rely on positional information to the point that they break-down when subjected to sentences with shifted position information. Specifically, when models are subjected to sentences starting from a non-zero position (excluding the effect of priming), they exhibit noticeably degraded performance on zero to full-shot tasks, across a range of model families and model sizes. Our findings raise questions about the efficacy of APEs to model the relativity of position information, and invite further introspection on the sentence and word order processing strategies employed by these models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑