语言模型的时间注意
Temporal Attention for Language Models
摘要:
基于Transformer架构的预训练语言模型在NLP中取得了巨大的成功。文本训练数据通常来自网络,因此带有特定时间的信息,但大多数语言模型忽略了这些信息。它们仅在文本数据上进行训练,限制了它们在时间上进行概括的能力。在这项工作中,我们扩展了Transformer架构的关键组件,即,自我注意力机制,并提出了时间注意-时间感知的自我注意力机制。时间注意力可以应用于任何Transformer模型,并且要求输入文本伴随其相关时间点。它允许Transformer捕获该时间信息并创建特定于时间的上下文化单词表示。我们利用这些表示的语义变化检测的任务,我们将我们提出的机制BERT和实验三个数据集在不同的语言(英语,德语和拉丁语),也不同的时间,大小和体裁。我们提出的模型在所有数据集上都取得了最先进的结果。
英文摘要:
Pretrained language models based on the transformer architecture have shown great success in NLP. Textual training data often comes from the web and is thus tagged with time-specific information, but most language models ignore this information. They are trained on the textual data alone, limiting their ability to generalize temporally. In this work, we extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention - a time-aware self-attention mechanism. Temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. It allows the transformer to capture this temporal information and create time-specific contextualized word representations. We leverage these representations for the task of semantic change detection; we apply our proposed mechanism to BERT and experiment on three datasets in different languages (English, German, and Latin) that also vary in time, size, and genre. Our proposed model achieves state-of-the-art results on all the datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。