SpacTor-T5:使用跨度破坏和替换 token 检测预训练 T5 模型
SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
AI总结:
本文提出 SpacTor 两阶段混合预训练流程,将 SC 与 RTD 结合用于 T5,在保持下游性能的同时减少 50% 迭代和 40% FLOPs,或在同等预算下提升基准表现。
摘要:
众所周知,预训练大型语言模型极其耗费资源,而且往往效率低下,未能充分利用训练文本序列中包含的信息。在本文中,我们提出了 SpacTor,这是一种新的训练流程,包括:(1)结合跨度破坏(SC)与 token 替换检测(RTD)的混合目标;(2)两阶段课程,在最初的 τ 次迭代中优化该混合目标,随后过渡到标准 SC 损失。我们通过实验表明,混合目标的有效性与两阶段预训练调度相关,并对其原因进行了广泛分析。在我们针对多种 NLP 任务、使用编码器-解码器架构(T5)开展的实验中,SpacTor-T5 取得了与标准 SC 预训练相同的下游性能,同时使预训练迭代次数减少 50%、总 FLOPs 降低 40%。换言之,在计算预算相同的情况下,我们发现 SpacTor 显著提升了下游基准性能。
英文摘要:
Pre-training large language models is known to be extremely resource intensive and often times inefficient, under-utilizing the information encapsulated in the training text sequences. In this paper, we present SpacTor, a new training procedure consisting of (1) a hybrid objective combining span corruption (SC) and token replacement detection (RTD), and (2) a two-stage curriculum that optimizes the hybrid objective over the initial $τ$ iterations, then transitions to standard SC loss. We show empirically that the effectiveness of the hybrid objective is tied to the two-stage pre-training schedule, and provide extensive analysis on why this is the case. In our experiments with encoder-decoder architectures (T5) on a variety of NLP tasks, SpacTor-T5 yields the same downstream performance as standard SC pre-training, while enabling a 50% reduction in pre-training iterations and 40% reduction in total FLOPs. Alternatively, given the same amount of computing budget, we find that SpacTor results in significantly improved downstream benchmark performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。