通过随机填充对位置嵌入进行极其简单有效的改进
A Frustratingly Easy Improvement for Position Embeddings via Random Padding
AI总结:
本文提出随机填充策略,通过微调时调整token顺序平衡位置嵌入更新次数,显著提升抽取式问答中答案位于后部位置的模型性能。
摘要:
位置嵌入编码了文本序列中token之间的位置关系,对基于Transformer的预训练语言模型中局部上下文特征的建模做出了巨大贡献。然而,在抽取式问答中,使用不同上下文长度的实例训练的位置嵌入可能并不如我们预期的那样表现良好。由于后部位置的嵌入更新次数少于前部位置嵌入,后部位置嵌入可能未得到充分训练。在本文中,我们提出了一种简单但有效的策略——随机填充,无需对现有预训练语言模型的架构进行任何修改。我们在微调时调整输入序列的token顺序,以平衡每个位置嵌入的更新次数。实验表明,随机填充能显著提升模型在答案位于后部位置的实例上的性能,尤其是当模型在短上下文上训练但在长上下文上评估时。我们的代码和数据将发布以供未来研究使用。
英文摘要:
Position embeddings, encoding the positional relationships among tokens in text sequences, make great contributions to modeling local context features in Transformer-based pre-trained language models. However, in Extractive Question Answering, position embeddings trained with instances of varied context lengths may not perform well as we expect. Since the embeddings of rear positions are updated fewer times than the front position embeddings, the rear ones may not be properly trained. In this paper, we propose a simple but effective strategy, Random Padding, without any modifications to architectures of existing pre-trained language models. We adjust the token order of input sequences when fine-tuning, to balance the number of updating times of every position embedding. Experiments show that Random Padding can significantly improve model performance on the instances whose answers are located at rear positions, especially when models are trained on short contexts but evaluated on long contexts. Our code and data will be released for future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。