arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02181cs.CLcs.AIcs.LG
收藏

大语言模型推理阶段并非所有层都必不可少

Not All Layers of LLMs Are Necessary During Inference

University of Electronic Science and Technology of China(电子科技大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)

Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, Zhongyuan Wang

AI总结:

针对大语言模型推理资源消耗高的问题,提出基于统计特征与经典分类器的AdaInfer算法,可自适应提前终止推理,在几乎不损失性能的前提下有效降低推理成本,且不影响模型跨任务泛化性。

摘要:

由于参数量庞大,大语言模型(Large Language Models, LLM)的推理阶段资源消耗极高。然而,提交给LLM的各类请求处理难度并不相同。通过分析我们发现,针对部分任务,LLM在某些中间层就能得出与最终输出相当的结果,也就是说LLM推理阶段并非所有层都必需。如果能预测推理结果在哪一层就能匹配(经所有层计算得到的)最终结果,就能大幅降低推理成本。为此,我们提出了一种简单有效的算法AdaInfer,可针对单个输入实例自适应终止推理流程。AdaInfer依托易于获取的统计特征与SVM等经典分类器实现。在Llama2系列、OPT等知名LLM上的实验表明,AdaInfer能达到平均17.8%的剪枝率,在情感分析任务上最高可达43%,且性能几乎无下降(<1%)。由于AdaInfer不会修改LLM参数,集成该算法的LLM仍能保持跨任务的泛化能力。

英文摘要:

Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑