掩码即所得：优化掩码语言建模以预训练BabyLMs
Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
AI总结：
针对2025年BabyLM挑战赛，提出自适应掩码概率的MLM改进方法，并引入子词嵌入，显著提升GLUE性能并超越基线。
摘要：
我们描述了针对2025年BabyLM挑战赛的策略。我们的主要贡献是一种改进的掩码语言建模（MLM）形式，它根据模型预测被掩码标记的能力来调整这些标记被掩码的概率。结果表明，在（Super）GLUE任务上，与标准MLM相比，性能有了显著提升。我们还引入了子词嵌入，发现这增强了模型的形态泛化能力。我们的提交在严格小规模赛道中超越了基线。
英文摘要：
We describe our strategy for the 2025 edition of the BabyLM Challenge. Our main contribution is that of an improved form of Masked Language Modeling (MLM), which adapts the probabilities of the tokens masked according to the model's ability to predict them. The results show a substantial increase in performance on (Super)GLUE tasks over the standard MLM. We also incorporate sub-token embeddings, finding that this increases the model's morphological generalization capabilities. Our submission beats the baseline in the strict-small track.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
