arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.00617cs.CL
收藏

面向低资源环境的混合 RNN-Transformer 架构高效语言建模

Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures

University of Bamberg(班贝格大学);University of Stuttgart(斯图加特大学)

Gabriel Lindenmaier, Sean Papay, Sebastian Padó

AI总结:

本文提出在 Transformer 中选择性替换注意力层为前馈和准循环神经网络层,以在低数据体制下提升性能并降低参数量,实验证明其在参数量更少时优于或媲美现有模型。

摘要:

基于 Transformer 的语言模型近期处于文本生成活跃研究的前沿。然而,这些模型的进步以高昂的训练成本为代价,参数数量达数十亿,计算需求以 petaflop/s-decades 衡量。本文通过选择性地将注意力层替换为前馈和准循环神经网络层,研究了基于 Transformer 的架构,以在低数据体制下提升模型性能。我们在标准的 Enwik8 和 Wikitext-103 语料库上测试了这些架构。结果表明,我们精简的架构在参数数量相当的情况下优于现有模型,并在大幅减少参数数量的同时,获得了与更大模型相当的性能。

英文摘要:

Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑