掩码预训练目标的任务复杂度如何影响下游性能？
How does the task complexity of masked pretraining objectives affect downstream performance?
AI总结：
本文探究掩码预训练目标的任务复杂度对下游性能的影响，发现更复杂的掩码目标能取得更好的下游结果，且至少需达到 MLM 一半的复杂度才能与之相当。
摘要：
掩码语言建模（MLM）是一种广泛使用的自监督预训练目标，其中模型需要根据上下文预测被掩码替换的原始 token。尽管更简单且计算高效的预训练目标（例如预测掩码 token 的首字符）最近显示出与 MLM 相当的结果，但没有任何带有掩码方案的目标在下游任务中优于 MLM。受其缺乏复杂性在性能下降中起关键作用这一假设的启发，我们验证了更复杂的掩码目标是否能取得更好的结果，并研究它们需要多少复杂性才能与 MLM 表现相当。我们使用 GLUE、SQuAD 和 Universal Dependencies 基准的实验结果表明，更复杂的目标倾向于展现更好的下游结果，且至少需要 MLM 复杂性的一半才能表现得与 MLM 相当。最后，我们讨论了如何从任务复杂度的角度使用掩码目标预训练模型。
英文摘要：
Masked language modeling (MLM) is a widely used self-supervised pretraining objective, where a model needs to predict an original token that is replaced with a mask given contexts. Although simpler and computationally efficient pretraining objectives, e.g., predicting the first character of a masked token, have recently shown comparable results to MLM, no objectives with a masking scheme actually outperform it in downstream tasks. Motivated by the assumption that their lack of complexity plays a vital role in the degradation, we validate whether more complex masked objectives can achieve better results and investigate how much complexity they should have to perform comparably to MLM. Our results using GLUE, SQuAD, and Universal Dependencies benchmarks demonstrate that more complicated objectives tend to show better downstream results with at least half of the MLM complexity needed to perform comparably to MLM. Finally, we discuss how we should pretrain a model using a masked objective from the task complexity perspective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
