arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.18838cs.CL
收藏

词序何时重要,何时不重要?

When does word order matter and when doesn't it?

McGill University(麦吉尔大学);Mila Quebec AI Institute(米拉魁北克人工智能研究所)

Xuanda Chen, Timothy O'Donnell, Siva Reddy

AI总结:

本文用语言冗余解释语言模型对词序扰动的不敏感,并通过未打乱与打乱句子间的 MI/PMI 量化词序信息量,揭示该效应在 SST-2、RTE 等任务间的差异。

摘要:

语言模型(LM)在自然语言理解(NLU)任务中可能表现得对词序变化不敏感。本文提出,语言冗余可以解释这一现象:词序与格标记等其他语言线索提供相互重叠、因而冗余的信息。我们的假设是,当词序提供冗余信息时,模型会对词序不敏感,而且不敏感程度因任务而异。我们利用未打乱句子与打乱句子之间的互信息(MI)来量化词序的信息量。结果表明,词序信息量越低,模型在未打乱和打乱句子之间的预测就越一致。我们还发现这种效应因任务而异:对于 SST-2 等某些任务,即使 Pointwise-MI(PMI)发生变化,LM 的预测也几乎总是与原始预测一致;而对于 RTE 等其他任务,当 PMI 降低时,一致性接近随机水平,也就是说词序确实非常重要。

英文摘要:

Language models (LMs) may appear insensitive to word order changes in natural language understanding (NLU) tasks. In this paper, we propose that linguistic redundancy can explain this phenomenon, whereby word order and other linguistic cues such as case markers provide overlapping and thus redundant information. Our hypothesis is that models exhibit insensitivity to word order when the order provides redundant information, and the degree of insensitivity varies across tasks. We quantify how informative word order is using mutual information (MI) between unscrambled and scrambled sentences. Our results show the effect that the less informative word order is, the more consistent the model's predictions are between unscrambled and scrambled sentences. We also find that the effect varies across tasks: for some tasks, like SST-2, LMs' prediction is almost always consistent with the original one even if the Pointwise-MI (PMI) changes, while for others, like RTE, the consistency is near random when the PMI gets lower, i.e., word order is really important.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑