从错误中学习：在语言预训练中使用错误预测作为伤害警报
Learning from Mistakes: Using Mis-predictions as Harm Alerts in Language Pre-Training
摘要：
在训练数据中拟合复杂模式，例如推理和常识，是语言预训练的关键挑战。根据最近的研究和我们的经验观察，一个可能的原因是训练数据中的一些容易拟合的模式，例如频繁共现的单词组合，主导和损害预训练，使模型难以拟合更复杂的信息。我们认为，错误的预测可以帮助定位这种占主导地位的模式，损害语言理解。当发生误预测时，应该存在与由模型拟合的导致误预测的误预测词频繁共现的模式。如果我们可以添加正则化来训练模型，以便在发生错误预测时更少地依赖于这种主导模式，并更多地关注其余更微妙的模式，则可以在预训练时有效地拟合更多信息。基于这种动机，我们提出了一种新的语言预训练方法，即错误预测作为伤害警报（MPA）。在MPA中，当在预训练期间发生错误预测时，我们使用其共现信息来指导自我注意模块的几个头部。Transformer模块中的一些自注意头部被优化，以将较低的注意权重分配给输入句子中经常与误预测同时出现的单词，而将较高的权重分配给其他单词。通过这样做，Transformer模型被训练为较少地依赖于具有误预测的主导频繁共现模式，而在误预测发生时更多地关注其余更复杂的信息。实验结果表明，MPA加快了BERT和ELECTRA的预训练，提高了它们在下游任务上的性能。
英文摘要：
Fitting complex patterns in the training data, such as reasoning and commonsense, is a key challenge for language pre-training. According to recent studies and our empirical observations, one possible reason is that some easy-to-fit patterns in the training data, such as frequently co-occurring word combinations, dominate and harm pre-training, making it hard for the model to fit more complex information. We argue that mis-predictions can help locate such dominating patterns that harm language understanding. When a mis-prediction occurs, there should be frequently co-occurring patterns with the mis-predicted word fitted by the model that lead to the mis-prediction. If we can add regularization to train the model to rely less on such dominating patterns when a mis-prediction occurs and focus more on the rest more subtle patterns, more information can be efficiently fitted at pre-training. Following this motivation, we propose a new language pre-training method, Mis-Predictions as Harm Alerts (MPA). In MPA, when a mis-prediction occurs during pre-training, we use its co-occurrence information to guide several heads of the self-attention modules. Some self-attention heads in the Transformer modules are optimized to assign lower attention weights to the words in the input sentence that frequently co-occur with the mis-prediction while assigning higher weights to the other words. By doing so, the Transformer model is trained to rely less on the dominating frequently co-occurring patterns with mis-predictions while focus more on the rest more complex information when mis-predictions occur. Our experiments show that MPA expedites the pre-training of BERT and ELECTRA and improves their performances on downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
