PolyLM:通过语言建模学习一词多义
PolyLM: Learning about Polysemy through Language Modeling
摘要:
为了避免词嵌入的“意义合并缺陷”,许多模型的目标是嵌入单个词义。这些方法曾经在词义归纳(WSI)等任务上表现良好,但后来被利用上下文嵌入的特定于任务的技术所取代。然而,意义嵌入和语境化并不一定是相互排斥的。我们引入PolyLM,一种将学习感觉嵌入的任务制定为语言建模问题的方法,允许应用上下文化技术。PolyLM基于关于词义的两个基本假设:首先,一个词在给定上下文中出现的概率等于其各个词义出现的概率之和;其次,对于一个词的给定出现,它的一个词义在上下文中往往比其他词义更合理。我们在WSI上对PolyLM进行了评估,结果表明它的性能比以前的意义嵌入技术要好得多,并且与当前最先进的专门WSI方法相匹配,尽管参数少了六倍。代码和预训练模型可在https://github.com/AlanAnsell/PolyLM上获得。
英文摘要:
To avoid the "meaning conflation deficiency" of word embeddings, a number of models have aimed to embed individual word senses. These methods at one time performed well on tasks such as word sense induction (WSI), but they have since been overtaken by task-specific techniques which exploit contextualized embeddings. However, sense embeddings and contextualization need not be mutually exclusive. We introduce PolyLM, a method which formulates the task of learning sense embeddings as a language modeling problem, allowing contextualization techniques to be applied. PolyLM is based on two underlying assumptions about word senses: firstly, that the probability of a word occurring in a given context is equal to the sum of the probabilities of its individual senses occurring; and secondly, that for a given occurrence of a word, one of its senses tends to be much more plausible in the context than the others. We evaluate PolyLM on WSI, showing that it performs considerably better than previous sense embedding techniques, and matches the current state-of-the-art specialized WSI method despite having six times fewer parameters. Code and pre-trained models are available at https://github.com/AlanAnsell/PolyLM.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。