arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.07295cs.CLcs.LG
收藏

在中间相遇:一种新的预训练范式

Meet in the Middle: A New Pre-training Paradigm

Microsoft Azure AI(微软Azure人工智能)

Anh Nguyen, Nikos Karampatziakis, Weizhu Chen

摘要:

大多数语言模型(LM)都是以自回归从左到右的方式训练和应用的,假设下一个标记只依赖于前一个标记。然而,这种假设忽略了在训练过程中使用完整序列信息的潜在好处,以及在推理过程中具有来自双方的上下文的可能性。在本文中,我们提出了一个新的预训练范例的技术,共同提高训练数据的效率和能力的LM在填充任务。第一个是训练目标,它将从左到右LM的预测与从右到左LM的预测对齐,在相同的数据上训练,但顺序相反。第二个是双向推理过程,使两个LM在中间相遇。我们通过对编程和自然语言模型的广泛实验,展示了我们的预训练范式的有效性,表现优于强大的基线。

英文摘要:

Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, assuming that the next token only depends on the preceding ones. However, this assumption ignores the potential benefits of using the full sequence information during training, and the possibility of having context from both sides during inference. In this paper, we propose a new pre-training paradigm with techniques that jointly improve the training data efficiency and the capabilities of the LMs in the infilling task. The first is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second is a bidirectional inference procedure that enables both LMs to meet in the middle. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models, outperforming strong baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑