使自回归模型能够填充掩码标记
Enabling Autoregressive Models to Fill In Masked Tokens
AI总结：
针对AR模型无法掩码填充、MLM模型计算低效的问题，提出MARIA架构，结合预训练MLM与AR模型的拼接隐藏状态训练线性解码器，使AR模型保留KV缓存推理优势的同时实现掩码填充，性能显著优于离散扩散模型。
摘要：
历史上，大型语言模型（LLMs）一直使用自回归（AR）或掩码语言建模（MLM）目标进行训练，近年来AR模型占据主导地位。然而，AR模型本质上无法进行掩码填充，即预测过去与未来上下文之间掩码标记的能力。相比之下，MLM模型在训练和推理期间都存在内在的计算低效问题，阻碍了其可扩展性。本工作引入MARIA（掩码与自回归填充架构），这是一种利用两种范式优势以实现最先进掩码填充性能的新方法。MARIA通过训练一个线性解码器来结合预训练的MLM和AR模型，该解码器将两者的拼接隐藏状态作为输入。这种最小化修改使AR模型能够执行填充任务，同时保留其在KV缓存加速推理方面的固有优势。我们的结果表明，在掩码填充任务上，MARIA显著优于现有方法（即离散扩散模型）。
英文摘要：
Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR models are inherently incapable of masked infilling, which is the ability to predict masked tokens between past and future context. In contrast, MLM models suffer from intrinsic computational inefficiencies during both training and inference that hinder their scalability. This work introduces MARIA (Masked and Autoregressive Infilling Architecture), a novel approach that leverages the strengths of both paradigms to achieve state-of-the-art masked infilling performance. MARIA combines a pre-trained MLM and AR model by training a linear decoder that takes their concatenated hidden states as input. This minimal modification enables the AR model to perform infilling while retaining its inherent advantages in terms of faster inference with KV caching. Our results demonstrate that MARIA significantly outperforms existing methods, namely discrete diffusion models, on masked infilling tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
