Brainformers：以复杂性换取效率
Brainformers: Trading Simplicity for Efficiency
AI总结：
本文提出 Brainformer 模块，通过采用包含稀疏门控、前馈和注意力等层的复杂结构替代传统统一骨干，在质量和效率上均超越现有密集和稀疏 Transformers，实现了更快的训练收敛与推理速度及更高的下游任务得分。
摘要：
Transformers 是近期自然语言处理和计算机视觉成功的关键。Transformers 拥有基本统一的骨干结构，层在前馈和自注意力之间交替，以构建深度网络。在此我们研究了这一设计选择，发现具有不同层原语排列的更复杂模块可以更高效。基于这一见解，我们开发了一个名为 Brainformer 的复杂模块，它由多种层组成，例如稀疏门控前馈层、密集前馈层、注意力层以及各种形式的层归一化和激活函数。在质量和效率方面，Brainformer 始终优于最先进的密集和稀疏 Transformers。一个每 token 具有 80 亿激活参数的 Brainformer 模型，与其 GLaM 对应模型相比，展现出 2 倍更快的训练收敛速度和 5 倍更快的步长时间。在下游任务评估中，与具有相似数量激活参数的 GLaM 相比，Brainformer 在微调后也展现出 3% 更高的 SuperGLUE 分数。最后，在少样本评估中，Brainformer 在每 token 计算量相似的情况下，大幅优于通过 NAS 导出的 Primer 密集模型。
英文摘要：
Transformers are central to recent successes in natural language processing and computer vision. Transformers have a mostly uniform backbone where layers alternate between feed-forward and self-attention in order to build a deep network. Here we investigate this design choice and find that more complex blocks that have different permutations of layer primitives can be more efficient. Using this insight, we develop a complex block, named Brainformer, that consists of a diverse sets of layers such as sparsely gated feed-forward layers, dense feed-forward layers, attention layers, and various forms of layer normalization and activation functions. Brainformer consistently outperforms the state-of-the-art dense and sparse Transformers, in terms of both quality and efficiency. A Brainformer model with 8 billion activated parameters per token demonstrates 2x faster training convergence and 5x faster step time compared to its GLaM counterpart. In downstream task evaluation, Brainformer also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM with a similar number of activated parameters. Finally, Brainformer largely outperforms a Primer dense model derived with NAS with similar computation per token on fewshot evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
