Next-token prediction is conventionally done using decoder-only Transformers with causal attention, as this approach allows for efficient reuse of keys and values. What if we were not compute-limited, should we still use decoder-only Transformers? In this work, we introduce Encoder-only Next Token Prediction (ENTP). We explore the differences between ENTP and decoder-only Transformers in expressive power and complexity, highlighting potential advantages of ENTP in settings with unbounded compute. We introduce the $\operatorname{Count3}$ task and show, both theoretically and experimentally, that while ENTP can perform this task easily, a decoder-only Transformer cannot. Finally, we empirically demonstrate the superior performance of ENTP across representative tasks where next-token prediction based Transformers can be evaluated, including addition, in-context learning, and language modeling.
Contextual morphologically-guided tokenization for Latin encoder models
Manning College of Information & Computer Sciences, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校信息与计算机科学学院);Institute for the Study of the Ancient World, New York University(纽约大学古代世界研究所)
ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Johns Hopkins University(约翰斯·霍普金斯大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);HLTCOE, Johns Hopkins University(约翰斯·霍普金斯大学语言与语音处理中心);NVIDIA(英伟达公司)
PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK(视觉、语音和信号处理中心(CVSSP),英国萨里大学);Surrey Institute for People-Centred AI, University of Surrey, Guildford, GU2 7XH, UK(萨里大学以人为中心的人工智能研究所,英国Guildford);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE(穆罕默德·本·扎耶德人工智能大学(MBZUAI),阿布扎赫德)
我们表明,Transformer编码器架构可以加快,有限的精度成本,通过用简单的线性变换“混合”输入令牌取代自我注意力子层。这些线性混合器,以及前馈层中的标准非线性,证明了在几个文本分类任务中建模语义关系的能力。最令人惊讶的是,我们发现用标准的非参数化傅立叶变换替换Transformer编码器中的自注意子层,在GLUE基准测试中实现了BERT对应物的92-97%的准确度,但在标准512输入长度下,GPU上的训练速度快了80%,TPU上的训练速度快了70%。在较长的输入长度下,我们的FNet模型明显更快:与Long Range Arena基准测试中的“高效”Transformers相比,FNet与最准确模型的准确性相匹配,同时在GPU上的所有序列长度(以及TPU上相对较短的长度)上都超过了最快的模型。最后,FNet的内存占用量很小,在较小的模型尺寸下特别有效;对于固定的速度和精度预算,小型FNet模型优于Transformer模型。
Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules
National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Hokkaido University(北海道大学);Institute of Artificial Intelligence, Institute of Dataspace, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院、数据空间研究院)
BrainG3N: A Dual-Purpose Tokenizer for Controllable 3D Brain MRI Generation
Department of Biomedical Data Science, Stanford University School of Medicine(斯坦福大学医学院生物医学数据科学系);Department of Mathematical Modelling, Statistics & Bioinformatics, Ghent University(根特大学数学建模、统计与生物信息学系);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系)