基于文本长度的置信度正则化掩码语言建模
Confidence Regularized Masked Language Modeling using Text Length
AI总结：
针对掩码语言建模在短文本上因忽略多义替代词而导致过度自信的问题，本文提出一种根据输入长度自适应调整强度的置信度正则化方法，并在GLUE和SQuAD上验证其同时提升准确率与校准性能。
摘要：
掩码语言建模是一种广泛使用的语言表示学习方法，模型在每次输入中预测一个被随机掩码的词。然而，这种方法在训练过程中通常只考虑单一正确答案，忽略了人类可能选择的多种合理替代方案。当输入文本较短时，这一问题变得更加突出，因为可能的词分布往往具有更高的熵，可能导致模型对其预测过于自信。为了缓解这一问题，我们提出了一种新颖的置信度正则化器，它根据输入长度自适应地调整正则化强度。在GLUE和SQuAD基准上的实验表明，我们的方法同时提高了准确率和期望校准误差。
英文摘要：
Masked language modeling is a widely used method for learning language representations, where the model predicts a randomly masked word in each input. However, this approach typically considers only a single correct answer during training, ignoring the variety of plausible alternatives that humans might choose. This issue becomes more pronounced when the input text is short, as the possible word distribution tends to have higher entropy, potentially causing the model to become overconfident in its predictions. To mitigate this, we propose a novel confidence regularizer that adaptively adjusts the regularization strength based on the input length. Experiments on the GLUE and SQuAD benchmarks show that our method improves both accuracy and expected calibration error
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
