arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06677cs.CL
收藏

混合模块(Mixture-of-Modules):将Transformer重塑为模块的动态组合体

Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules

Peking University(北京大学);Renmin University(中国人民大学);Tsinghua University(清华大学);Ant Group(蚂蚁集团)

Zhuocheng Gong, Ang Lv, Jian Guan, Junxi Yan, Wei Wu, Huishuai Zhang, Minlie Huang, Dongyan Zhao, Rui Yan

AI总结:

本研究提出混合模块(MoM)架构,打破Transformer按深度顺序计算的传统,通过路由器动态选择模块处理token,在OpenWebText预训练下,不同参数量的MoM均优于 vanilla Transformer,且能在性能相当的前提下显著降低计算量与内存占用。

摘要:

在Transformer中,是否始终有必要按照从浅层到深层的顺序计算token? vanilla Transformer及其变体的持续成功表明答案无疑是肯定的。然而在本研究中,我们尝试打破这种按深度排序的传统,提出了一种名为混合模块(mixture-of-modules, MoM)的新型架构,其核心直觉是:任何层无论处于什么位置,只要具备所需的处理能力,都可用于计算token。\nMoM的构建始于一组由多头注意力和前馈网络定义的有限模块集合,每个模块都有独特的参数化方式。随后两个路由器会从集合中迭代选择注意力模块和前馈模块来处理单个token,这种选择会在token的前向传播过程中动态扩展计算图,最终形成一个模块组合体。我们证明,MoM不仅为Transformer及其众多变体提供了统一框架,还为减少Transformer参数化中的冗余提供了一种灵活且可学习的方法。\n我们使用OpenWebText预训练了多种MoM模型。实验结果表明,不同参数量的MoM在GLUE和XSUM基准测试上的表现始终优于vanilla Transformer。更值得注意的是,在固定参数预算下,MoM-large的计算图深度相比GPT-2-large提升了38%以上,在GLUE上带来1.4的绝对提升,在XSUM上带来1的绝对提升。另一方面,MoM-large还可将深度降低60%以上,同时每层包含更多模块,相比GPT-2-large减少16%的TFLOPs和43%的内存占用,且性能保持相当。

英文摘要:

Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted "yes". In this work, however, we attempt to break the depth-ordered convention by proposing a novel architecture dubbed mixture-of-modules (MoM), which is motivated by an intuition that any layer, regardless of its position, can be used to compute a token as long as it possesses the needed processing capabilities. The construction of MoM starts from a finite set of modules defined by multi-head attention and feed-forward networks, each distinguished by its unique parameterization. Two routers then iteratively select attention modules and feed-forward modules from the set to process a token. The selection dynamically expands the computation graph in the forward pass of the token, culminating in an assembly of modules. We show that MoM provides not only a unified framework for Transformers and their numerous variants but also a flexible and learnable approach for reducing redundancy in Transformer parameterization. We pre-train various MoMs using OpenWebText. Empirical results demonstrate that MoMs, of different parameter counts, consistently outperform vanilla transformers on both GLUE and XSUM benchmarks. More interestingly, with a fixed parameter budget, MoM-large enables an over 38% increase in depth for computation graphs compared to GPT-2-large, resulting in absolute gains of 1.4 on GLUE and 1 on XSUM. On the other hand, MoM-large also enables an over 60% reduction in depth while involving more modules per layer, yielding a 16% reduction in TFLOPs and a 43% decrease in memory usage compared to GPT-2-large, while maintaining comparable performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑