学习更好的掩码策略以实现更好的语言模型预训练
Learning Better Masking for Better Language Model Pre-training
AI总结：
本文提出两种时间变化的调度掩码方法，在不同训练阶段自适应调整掩码比例与内容，从而提升掩码语言模型预训练的效率和下游任务效果。
摘要：
掩码语言建模（MLM）已被广泛用作预训练语言模型（PrLMs）中的去噪目标。现有的PrLMs通常采用随机令牌掩码策略，即在整个训练过程中应用固定的掩码比例，并以相等的概率掩码不同的内容。然而，模型可能会受到预训练状态的复杂影响，而这种状态会随着训练时间的推移而相应变化。在本文中，我们表明，这种在掩码比例和掩码内容上时间不变的MLM设置不太可能带来最优结果，这促使我们探索时间变化的MLM设置的影响。我们提出了两种调度掩码方法，在不同的训练阶段自适应地调整掩码比例和掩码内容，这在下游任务上验证了预训练效率和效果的提升。我们的工作是对掩码比例和内容的时间变化掩码策略的先驱性研究，并更好地理解了掩码比例和掩码内容如何影响MLM预训练。
英文摘要：
Masked Language Modeling (MLM) has been widely used as the denoising objective in pre-training language models (PrLMs). Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training. However, the model may receive complicated impact from pre-training status, which changes accordingly as training time goes on. In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings. We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks. Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pre-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
