arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.01705cs.LGcs.AI
收藏

信息通路假说:Transformers 是动态自集成模型

The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles

Rensselaer Polytechnic Institute(伦斯勒理工学院);IBM T. J. Watson Research Center(IBM托马斯·J·沃森研究中心)

Md Shamim Hussain, Mohammed J. Zaki, Dharmashankar Subramanian

AI总结:

提出信息通路假说与随机子采样自注意力 SSA,通过可预测的稀疏通路训练 transformer,降低 4–8 倍自注意力成本并提升泛化。

摘要:

Transformers 采用稠密自注意力机制,该机制为长程连接提供了很大的灵活性。在深层 transformer 的多个层中,可能的连接模式数量呈指数级增长。然而,其中只有极少数模式有助于网络性能,而真正必不可少的模式更少。我们假设,transformer 内部存在稀疏连接的子网络,称为信息通路,并且这些子网络可以被独立训练。不过,这些通路的动态(即依赖于输入的)性质使其难以在训练期间对稠密自注意力进行剪枝。但这些通路的整体分布通常是可预测的。我们利用这一事实提出了随机子采样自注意力(Stochastically Subsampled self-Attention,SSA)——一种面向 transformers 的通用训练策略,能够在训练期间将自注意力的内存和计算成本均降低 4 到 8 倍,同时还可作为一种正则化方法,相比稠密训练改善泛化能力。我们表明,可以由网络内部经过子采样的通路构成一个子模型集成,并且该集成能够取得优于其稠密注意力对应模型的性能。我们在自然语言处理、计算机视觉和图学习等多种任务上进行了实验,涵盖生成式和判别式设置,以为我们的主张提供经验证据,并展示所提方法的有效性。

英文摘要:

Transformers use the dense self-attention mechanism which gives a lot of flexibility for long-range connectivity. Over multiple layers of a deep transformer, the number of possible connectivity patterns increases exponentially. However, very few of these contribute to the performance of the network, and even fewer are essential. We hypothesize that there are sparsely connected sub-networks within a transformer, called information pathways which can be trained independently. However, the dynamic (i.e., input-dependent) nature of these pathways makes it difficult to prune dense self-attention during training. But the overall distribution of these pathways is often predictable. We take advantage of this fact to propose Stochastically Subsampled self-Attention (SSA) - a general-purpose training strategy for transformers that can reduce both the memory and computational cost of self-attention by 4 to 8 times during training while also serving as a regularization method - improving generalization over dense training. We show that an ensemble of sub-models can be formed from the subsampled pathways within a network, which can achieve better performance than its densely attended counterpart. We perform experiments on a variety of NLP, computer vision and graph learning tasks in both generative and discriminative settings to provide empirical evidence for our claims and show the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑