arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.08561cs.CL
收藏

预训练Transformers作为基于能量的完形填空模型

Pre-Training Transformers as Energy-Based Cloze Models

Stanford University(斯坦福大学);Google Brain(谷歌大脑)

Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning

摘要:

我们介绍了Electric,一个基于能量的完形填空模型,用于文本上的表示学习。像BERT一样,它是一个给定上下文的令牌的条件生成模型。但是,Electric不使用掩码或输出上下文中可能发生的令牌的完整分布。相反,它为每个输入标记分配一个标量能量分数,指示它在给定上下文的情况下的可能性。我们训练电动使用基于噪声对比估计的算法,并阐明这种学习目标是如何密切相关的最近提出的ELECTRA预训练方法。Electric在转移到下游任务时表现良好,并且在产生文本的可能性得分方面特别有效:它比语言模型更好地重新排列语音识别n-最佳列表,并且比掩蔽的语言模型快得多。此外,它提供了一个更清晰,更有原则的观点,什么ELECTRA学习在预培训。

英文摘要:

We introduce Electric, an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context. We train Electric using an algorithm based on noise-contrastive estimation and elucidate how this learning objective is closely related to the recently proposed ELECTRA pre-training method. Electric performs well when transferred to downstream tasks and is particularly effective at producing likelihood scores for text: it re-ranks speech recognition n-best lists better than language models and much faster than masked language models. Furthermore, it offers a clearer and more principled view of what ELECTRA learns during pre-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑