当注意力遇到快速递归：用简化的计算训练语言模型
When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
摘要：
由于计算时间和成本的增长，大语言模型越来越难以训练。在这项工作中，我们提出了SRU++，一个高效的体系结构，结合了快速递归和序列建模的关注。SRU++具有强大的建模能力和训练效率。在标准语言建模任务（如Enwik 8，Wiki-103和Billion Word数据集）上，我们的模型获得了更好的每字符位数和困惑度，同时与性能最好的Transformer模型相比，使用的训练成本减少了3 - 10倍。例如，我们的模型在Enwik 8数据集上实现了最先进的结果，使用8 GPU机器上的1.6天训练。我们进一步证明，SRU++需要最少的关注，接近国家的最先进的性能。我们的研究结果表明，联合利用快速递归，很少关注，这是加速模型训练和推理的一个有前途的方向。
英文摘要：
Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and Billion Word datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to top-performing Transformer models. For instance, our model achieves a state-of-the-art result on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
