神经语言模型中的拼写约定敏感性
Spelling convention sensitivity in neural language models
摘要:
我们研究了在非常大的各种英语文本集合上训练的大型神经语言模型是否学习了英国与美国拼写惯例的潜在长距离依赖性,即,拼写在模型生成的字符串中是否始终是一个或另一个。与非表面底层结构中的长距离依赖性(例如,语法),拼写一致性更容易在LM和用于训练它们的文本语料库中测量,这可以提供对某些观察到的模型行为的额外洞察。使用一组独特的探测词无论是英国或美国英语,我们首先建立培训语料库表现出实质性的(虽然不是全部)的一致性。一个大型的T5语言模型似乎确实内化了这种一致性,尽管只针对观察到的词汇项(而不是具有英国/美国拼写模式的临时词)。我们进一步实验通过在已去偏的合成数据上微调T5来校正训练数据中的偏置,并发现微调后的T5对拼写一致性仅保持一定程度的敏感。进一步的实验表明GPT 2也受到类似的限制。
英文摘要:
We examine whether large neural language models, trained on very large collections of varied English text, learn the potentially long-distance dependency of British versus American spelling conventions, i.e., whether spelling is consistently one or the other within model-generated strings. In contrast to long-distance dependencies in non-surface underlying structure (e.g., syntax), spelling consistency is easier to measure both in LMs and the text corpora used to train them, which can provide additional insight into certain observed model behaviors. Using a set of probe words unique to either British or American English, we first establish that training corpora exhibit substantial (though not total) consistency. A large T5 language model does appear to internalize this consistency, though only with respect to observed lexical items (not nonce words with British/American spelling patterns). We further experiment with correcting for biases in the training data by fine-tuning T5 on synthetic data that has been debiased, and find that finetuned T5 remains only somewhat sensitive to spelling consistency. Further experiments show GPT2 to be similarly limited.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。