ExLM：重新思考掩码语言模型中 [MASK] 标记的影响
ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
AI总结：
本文分析 [MASK] 标记导致的损坏语义问题，提出通过扩展掩码状态并建模其依赖关系来增强上下文的 ExLM，在文本与 SMILES 建模上均显著提升性能。
摘要：
掩码语言模型（MLM）已在许多自监督表示学习任务中取得显著成功。MLM 的训练方式是使用 [MASK] 标记随机掩码输入序列的一部分，并学习基于剩余上下文重建原始内容。本文探究了 [MASK] 标记对 MLM 的影响。分析研究表明，掩码标记会引入损坏语义问题，即损坏的上下文可能传达多种含糊含义。该问题也是影响 MLM 在下游任务上性能的关键因素。基于这些发现，我们提出了一种新颖的增强上下文 MLM，即 ExLM。我们的方法在输入上下文中扩展 [MASK] 标记，并对这些扩展状态之间的依赖关系进行建模。这一增强提高了上下文容量，使模型能够捕获更丰富的语义信息，从而在预训练期间有效缓解损坏语义问题。实验结果表明，ExLM 在文本建模和 SMILES 建模任务中均取得显著性能提升。进一步分析证实，ExLM 通过上下文增强丰富了语义表示，并有效减少了 MLM 中常见的语义多模态现象。
英文摘要：
Masked Language Models (MLMs) have achieved remarkable success in many self-supervised representation learning tasks. MLMs are trained by randomly masking portions of the input sequences with [MASK] tokens and learning to reconstruct the original content based on the remaining context. This paper explores the impact of [MASK] tokens on MLMs. Analytical studies show that masking tokens can introduce the corrupted semantics problem, wherein the corrupted context may convey multiple, ambiguous meanings. This problem is also a key factor affecting the performance of MLMs on downstream tasks. Based on these findings, we propose a novel enhanced-context MLM, ExLM. Our approach expands [MASK] tokens in the input context and models the dependencies between these expanded states. This enhancement increases context capacity and enables the model to capture richer semantic information, effectively mitigating the corrupted semantics problem during pre-training. Experimental results demonstrate that ExLM achieves significant performance improvements in both text modeling and SMILES modeling tasks. Further analysis confirms that ExLM enriches semantic representations through context enhancement, and effectively reduces the semantic multimodality commonly observed in MLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
