arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.20022cs.CLcs.LG
收藏

仅解码器 Transformer 中的动态层选择

Dynamic layer selection in decoder-only transformers

McGill University(麦吉尔大学);Amazon Science(亚马逊科学);Mila Québec AI Institute(魁北克米拉人工智能研究所)

Theodore Glavas, Joud Chataoui, Florence Regol, Wassim Jabbour, Antonios Valkanas, Boris N. Oreshkin, Mark Coates

AI总结:

本文比较仅解码器大模型的跳层与提前退出动态推理,发现跳层更鲁棒,并通过 oracle 控制器证明逐序列层选择平均仅用 23.3% 的层即可保持完整模型性能。

摘要:

大型语言模型(LLM)的庞大规模推动了对推理优化的探索。一种有效方法是动态推理,即根据当前样本调整架构,以降低总体计算成本。我们通过实验考察了自然语言生成(NLG)中两种常见的动态推理方法:跳层和提前退出。我们发现,与提前退出相比,预训练的仅解码器模型对通过跳层进行的层移除具有显著更强的鲁棒性。我们证明了在逐 token 基础上利用隐藏状态信息为跳层调整计算的难度。最后,我们通过构建一个 oracle 控制器表明,逐序列的动态计算分配有望带来显著的效率提升。值得注意的是,我们发现存在一种分配方式,平均仅使用完整模型 23.3% 的层即可达到与其相同的性能。

英文摘要:

The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑