在单轮语言模型预训练中丢弃 Dropout
Drop Dropout on Single-Epoch Language Model Pretraining
AI总结：
本文通过实验发现在BERT和Pythia等语言模型的单轮预训练中不使用Dropout能提升下游任务性能和模型可编辑性，提倡在单轮预训练中丢弃Dropout。
摘要：
最初，Dropout 被视为一种突破性的正则化技术，通过减少过拟合在几乎所有深度学习应用中降低了过拟合并提升了性能。然而，现代 LLM 常见的单轮预训练任务产生的过拟合极少，导致大型 LLM 不再使用 Dropout。尽管如此，尚未有人对 Dropout 在 LM 预训练中的作用进行彻底的实证调查。通过在不同 Dropout 水平下对掩码（BERT）和自回归（Pythia 160M 和 1.4B）LM 进行单轮预训练实验，我们发现在预训练期间不应用 Dropout 时，语言建模、形态句法（BLiMP）、问答（SQuAD）和自然语言推理（MNLI）的下游性能均有所提升。我们还发现，最近引入的“早期 Dropout”与完全不应用 Dropout 相比，同样会降低性能。我们进一步研究了模型的可编辑性，发现无 Dropout 训练的模型在基于梯度的模型编辑（MEND）中更成功，在基于表示的模型编辑（ReFT）中表现相当。因此，我们提倡在单轮预训练中丢弃 Dropout。
英文摘要：
Originally, dropout was seen as a breakthrough regularization technique that reduced overfitting and improved performance in almost all applications of deep learning by reducing overfitting. Yet, single-epoch pretraining tasks common to modern LLMs yield minimal overfitting, leading to dropout not being used for large LLMs. Nevertheless, no thorough empirical investigation has been done on the role of dropout in LM pretraining. Through experiments in single-epoch pretraining of both masked (BERT) and autoregressive (Pythia 160M and 1.4B) LMs with varying levels of dropout, we find that downstream performance in language modeling, morpho-syntax (BLiMP), question answering (SQuAD), and natural-language inference (MNLI) improves when dropout is not applied during pretraining. We additionally find that the recently-introduced "early dropout" also degrades performance over applying no dropout at all. We further investigate the models' editability, and find that models trained without dropout are more successful in gradient-based model editing (MEND) and equivalent in representation-based model editing (ReFT). Therefore, we advocate to drop dropout during single-epoch pretraining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
