arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.12819cs.LGcs.AI
收藏

面向参数高效 Transformer 的动态层绑定

Dynamic Layer Tying for Parameter-Efficient Transformers

Blavatnik School of Computer Science, Tel Aviv University(特拉维夫大学布拉瓦特尼克计算机科学学院)

Tamir David Hay, Lior Wolf

AI总结:

本文提出用强化学习在训练中动态决定 transformer 各层独立训练或复制先前层权重,以实现层绑定与权重共享,在略降困惑度优势下大幅削减参数和训练内存。

摘要:

为减少深度 transformer 网络中可训练参数的数量,我们采用强化学习在训练过程中动态选择层并将它们绑定。每隔若干次迭代,RL 智能体需要决定:是独立训练每个层 i,还是复制先前某个层 j<i 的权重。这促进了权重共享,减少了可训练参数数量,同时也是一种有效的正则化技术。实验评估证实,在困惑度方面,我们的模型略优于基线 transformer 模型,并大幅减少了可训练参数数量。特别是,训练期间的内存消耗比传统训练方法最多降低一个数量级。

英文摘要:

In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j<i$. This facilitates weight sharing, reduces the number of trainable parameters, and also serves as an effective regularization technique. Experimental evaluations validate that our model modestly outperforms the baseline transformer model with regard to perplexity and drastically reduces the number of trainable parameters. In particular, the memory consumption during training is up to one order of magnitude less than the conventional training method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑