SHAQ：具有准递归的单头注意
SHAQ: Single Headed Attention with Quasi-Recurrence
摘要：
自然语言处理的研究最近一直由大规模的Transformer模型主导。虽然他们在许多重要的语言任务上达到了最先进的水平，但Transformers通常需要昂贵的计算资源，并需要数天到数周的训练。这对大型科技公司和顶尖研究型大学的研究人员来说是可行的，但对斗志旺盛的初创企业创始人、学生和独立研究人员来说就不可行了。Stephen Merity的SHA-RNN是一种紧凑的混合注意力-RNN模型，专为消费级建模而设计，因为它需要更少的参数和更少的训练时间来达到接近最先进的结果。我们分析Merity的模型在这里通过探索性模型分析几个单位的架构，同时考虑到我们的评估培训时间和整体质量。最终，我们将这些发现结合到一个新的架构中，我们称之为SHAQ：单头注意准递归神经网络。通过我们的新架构，我们实现了与SHA-RNN相似的准确性结果，同时在训练中实现了4倍的速度提升。
英文摘要：
Natural Language Processing research has recently been dominated by large scale transformer models. Although they achieve state of the art on many important language tasks, transformers often require expensive compute resources, and days spanning to weeks to train. This is feasible for researchers at big tech companies and leading research universities, but not for scrappy start-up founders, students, and independent researchers. Stephen Merity's SHA-RNN, a compact, hybrid attention-RNN model, is designed for consumer-grade modeling as it requires significantly fewer parameters and less training time to reach near state of the art results. We analyze Merity's model here through an exploratory model analysis over several units of the architecture considering both training time and overall quality in our assessment. Ultimately, we combine these findings into a new architecture which we call SHAQ: Single Headed Attention Quasi-recurrent Neural Network. With our new architecture we achieved similar accuracy results as the SHA-RNN while accomplishing a 4x speed boost in training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
