用于MLM预训练的动态掩码率调度
Dynamic Masking Rate Schedules for MLM Pretraining
AI总结：
针对BERT等掩码语言模型固定15%掩码率的局限，提出动态调度掩码率的方法。在预训练中线性降低掩码率不仅提升了GLUE准确率，还实现了最高1.89倍的加速和帕累托改进。
摘要：
大多数关于使用掩码语言建模（MLM）目标训练的transformers的研究，都采用原始BERT模型15%的固定掩码率。我们提出在整个训练过程中动态调度掩码率。我们发现，在预训练过程中线性降低掩码率，与固定掩码率基线相比，BERT-base和BERT-large的平均GLUE准确率分别提升了0.46%和0.25%。这些提升得益于模型同时接触高掩码率和低掩码率状态，从而获得了两种设置的共同优势。结果表明，掩码率调度是提升掩码语言模型质量的一种简单方法，它不仅为BERT-base实现了高达1.89倍的预训练加速，还为BERT-large带来了帕累托改进。
英文摘要：
Most works on transformers trained with the Masked Language Modeling (MLM) objective use the original BERT model's fixed masking rate of 15%. We propose to instead dynamically schedule the masking rate throughout training. We find that linearly decreasing the masking rate over the course of pretraining improves average GLUE accuracy by up to 0.46% and 0.25% in BERT-base and BERT-large, respectively, compared to fixed rate baselines. These gains come from exposure to both high and low masking rate regimes, providing benefits from both settings. Our results demonstrate that masking rate scheduling is a simple way to improve the quality of masked language models, achieving up to a 1.89x speedup in pretraining for BERT-base as well as a Pareto improvement for BERT-large.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
