arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.03327cs.CL
收藏

可调延迟的Transformer编码器用于语言理解

Latency Adjustable Transformer Encoder for Language Understanding

Sharif University of Technology(谢里夫理工大学)

Sajjad Kachuee, Mohammad Sharifkhani

AI总结:

提出一种基于注意力上下文贡献度度量的Transformer编码器,在微调阶段移除低贡献词向量,实现推理延迟可调,将Llama3的TTFT提升至2.9倍。

摘要:

调整自然语言理解模型的延迟、功耗和准确性是高效架构的理想目标。本文提出了一种高效的Transformer架构,能够根据期望的推理延迟加速比自适应地调整推理计算成本。在微调阶段,所提出的方法使用一种名为注意力上下文贡献(ACC)的度量来检测不太重要的隐藏序列元素(词向量),并在每个编码器层中消除它们。微调阶段之后,凭借新颖的离线调优特性,模型的推理延迟可以在广泛的推理加速比选择范围内进行调整,而无需任何进一步训练。大量实验表明,较高Transformer层中的大多数词向量对后续层的贡献较小,因此可以移除它们以提高推理延迟。在各种语言理解、文本生成和指令调优任务及基准上的实验结果表明,该方法在多样数据集上的有效性,且对输入的全局上下文影响最小。该技术将Llama3的首词生成时间(TTFT)最多提升2.9倍,同时性能下降很小。所提出的方法认为,在大语言模型(LLMs)中,尽管完整网络对于训练是必要的,但在微调阶段可以对其进行截断。

英文摘要:

Adjusting the latency, power, and accuracy of natural language understanding models is a desirable objective of an efficient architecture. This paper proposes an efficient Transformer architecture that adjusts the inference computational cost adaptively with a desired inference latency speedup. In fine-tuning phase, the proposed method detects less important hidden sequence elements (word-vectors) and eliminates them in each encoder layer using a proposed Attention Context Contribution (ACC) metric. After the fine-tuning phase, with the novel offline-tuning property, the inference latency of the model can be adjusted in a wide range of inference speedup selections without any further training. Extensive experiments reveal that most word-vectors in higher Transformer layers contribute less to subsequent layers, allowing their removal to improve inference latency. Experimental results on various language understanding, text generation, and instruction tuning tasks and benchmarks demonstrate the approach's effectiveness across diverse datasets, with minimal impact on the input's global context. The technique improves Time-to-First-Token (TTFT) of Llama3 by up to 2.9x, with minor performance drop. The suggested approach posits that in Large Language Models (LLMs), although the complete network is necessary for training, it can be truncated during the fine-tuning phase.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑