arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.04657cs.CL
收藏

Transformer 头部的可微子集剪枝

Differentiable Subset Pruning of Transformer Heads

ETH Zürich(苏黎世联邦理工学院);University of Cambridge(剑桥大学)

Jiaoda Li, Ryan Cotterell, Mrinmaya Sachan

AI总结:

本文提出一种名为可微子集剪枝的 Transformer 头部剪枝新方法,通过随机梯度下降学习每个头的重要性变量并施加硬约束以控制未剪枝头数量,在自然语言推理和机器翻译任务中表现优于以往工作且能精确控制稀疏度。

摘要:

多头注意力是若干独立关注输入不同部分的注意力机制的集合,它是 Transformer 的关键组成部分。然而,近期研究表明,Transformer 多头注意力机制中很大比例的头可以被安全地剪枝掉,而不会显著损害模型性能;这种剪枝使得模型在实际应用中明显更小且更快。我们的工作引入了一种新的头部剪枝技术,我们称之为可微子集剪枝(differentiable subset pruning)。直观上,我们的方法学习每个头的重要性变量,然后对未剪枝头的数量施加用户指定的硬约束。重要性变量通过随机梯度下降进行学习。我们在自然语言推理和机器翻译上进行了实验;结果表明,可微子集剪枝的表现与先前工作相当或更好,同时能精确控制稀疏度水平。

英文摘要:

Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in a Transformer's multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model; such pruning leads to models that are noticeably smaller and faster in practice. Our work introduces a new head pruning technique that we term differentiable subset pruning. Intuitively, our method learns per-head importance variables and then enforces a user-specified hard constraint on the number of unpruned heads. The importance variables are learned via stochastic gradient descent. We conduct experiments on natural language inference and machine translation; we show that differentiable subset pruning performs comparably or better than previous works while offering precise control of the sparsity level.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑