arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.07003cs.CLcs.LG
收藏

长度自适应Transformer:训练一次长度下降,随时使用搜索

Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

Clova AI, NAVER Corp.(NAVER公司Clova AI);New York University(纽约大学)

Gyuwan Kim, Kyunghyun Cho

摘要:

尽管Transformers具有令人印象深刻的精度,但其计算成本通常在有限的计算资源下无法使用。大多数以前的方法,以提高推理效率需要一个单独的模型,为每一个可能的计算预算。在本文中,我们扩展了POWER-BERT(Goyal等人,2020),并提出了长度自适应Transformer,可用于各种推理场景后,一次性训练。我们用LengthDrop(dropout的一种结构变体)训练一个Transformer,它随机确定每一层的序列长度。然后,我们进行多目标的进化搜索,找到一个长度配置,最大限度地提高精度,并在任何给定的计算预算的效率指标最小化。此外,我们显着扩展的POWER-BERT的适用性超出序列级分类到标记级分类与丢弃和恢复过程,暂时丢弃字向量在中间层和恢复在最后一层,如果必要的。我们实证验证所提出的方法的效用,通过展示优越的准确性和效率的权衡下,各种设置,包括跨度为基础的问答和文本分类。代码可在https://github.com/clovaai/length-adaptive-transformer上获得。

英文摘要:

Despite transformers' impressive accuracy, their computational cost is often prohibitive to use with limited computational resources. Most previous approaches to improve inference efficiency require a separate model for each possible computational budget. In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training. We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer. We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget. Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into token-level classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary. We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification. Code is available at https://github.com/clovaai/length-adaptive-transformer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑