arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.01600cs.LGcs.CL
收藏

ENTP:仅编码器的下一词元预测

ENTP: Encoder-only Next Token Prediction

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Korea University(高丽大学)

Ethan Ewer, Daewon Chae, Thomas Zeng, Jinkyu Kim, Kangwook Lee

AI总结:

本文提出仅编码器下一词元预测(ENTP),通过理论分析与实验证明其在表达能力和多项任务上优于仅解码器 Transformer。

摘要:

下一词元预测通常使用具有因果注意力的仅解码器 Transformer 来完成,因为这种方法可以高效地重用键和值。如果我们不受计算限制,我们是否仍应使用仅解码器 Transformer?在这项工作中,我们提出了仅编码器下一词元预测(ENTP)。我们探讨了 ENTP 与仅解码器 Transformer 在表达能力和复杂度上的差异,强调了 ENTP 在计算不受限环境中的潜在优势。我们引入了 $\operatorname{Count3}$ 任务,并从理论和实验两方面证明,虽然 ENTP 可以轻松完成该任务,但仅解码器 Transformer 无法完成。最后,我们通过实验证明了 ENTP 在可评估基于下一词元预测的 Transformer 的代表性任务上的优越性能,这些任务包括加法、上下文学习和语言建模。

英文摘要:

Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only Transformers? In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP in settings with unbounded compute. We introduce the $\operatorname{Count3}$ task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate the superior performance of ENTP across representative tasks where next-token prediction based Transformers can be evaluated, including addition, in-context learning, and language modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑