Not All Layers of LLMs Are Necessary During Inference
University of Electronic Science and Technology of China(电子科技大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
由于参数量庞大,大语言模型(Large Language Models, LLM)的推理阶段资源消耗极高。然而,提交给LLM的各类请求处理难度并不相同。通过分析我们发现,针对部分任务,LLM在某些中间层就能得出与最终输出相当的结果,也就是说LLM推理阶段并非所有层都必需。如果能预测推理结果在哪一层就能匹配(经所有层计算得到的)最终结果,就能大幅降低推理成本。为此,我们提出了一种简单有效的算法AdaInfer,可针对单个输入实例自适应终止推理流程。AdaInfer依托易于获取的统计特征与SVM等经典分类器实现。在Llama2系列、OPT等知名LLM上的实验表明,AdaInfer能达到平均17.8%的剪枝率,在情感分析任务上最高可达43%,且性能几乎无下降(<1%)。由于AdaInfer不会修改LLM参数,集成该算法的LLM仍能保持跨任务的泛化能力。
英文摘要:
Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. However, not all requests posed to LLMs are equally difficult to handle. Through analysis, we show that for some tasks, LLMs can achieve results comparable to the final output at some intermediate layers. That is, not all layers of LLMs are necessary during inference. If we can predict at which layer the inferred results match the final results (produced by evaluating all layers), we could significantly reduce the inference cost. To this end, we propose a simple yet effective algorithm named AdaInfer to adaptively terminate the inference process for an input instance. AdaInfer relies on easily obtainable statistical features and classic classifiers like SVM. Experiments on well-known LLMs like the Llama2 series and OPT, show that AdaInfer can achieve an average of 17.8% pruning ratio, and up to 43% on sentiment tasks, with nearly no performance drop (<1%). Because AdaInfer does not alter LLM parameters, the LLMs incorporated with AdaInfer maintain generalizability across tasks.
Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment
Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Zhejiang University(浙江大学);Beijing Institute for General Artificial Intelligence(北京一般人工智能研究所)
Humans and LLMs Diverge on Probabilistic Inferences
McGill University(麦吉尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);University of Vienna(维也纳大学);FNRS – UCLouvain(佛罗伦萨-鲁汶大学);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席)
Bitnet.cpp: Efficient Edge Inference for Ternary LLMs
Peking University(北京大学);University of Science and Technology of China(中国科学技术大学);University of Chinese Academy of Sciences(中国科学院大学);Microsoft Research(微软研究院)
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
SINTEF(辛泰夫(挪威研究机构));Singapore Management University(新加坡管理大学);Oslo Metropolitan University(奥斯陆城市大学);Kristiania University of Applied Sciences(克里斯蒂安尼亚应用科技大学)
本研究在Raspberry Pi 4上评估了28个量化LLM在能效、推理性能和准确率上的权衡,通过硬件级能耗分析与多数据集基准测试,为资源受限环境下的可持续LLM部署提供了可操作的见解。
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);Alibaba Cloud Computing(阿里云计算);Xiaohongshu Inc.(小红书公司);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steering
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);College of Cryptology and Cyber Science, Nankai University(南开大学密码与网络科学学院)
Context-Aware Membership Inference Attacks against Pre-trained Large Language Models
Mohamed bin Zayed University of Artificial Intelligence(莫扎伊德·本·扎耶德人工智能大学);Brave Software(Brave软件公司);Imperial College London(帝国理工学院);National University of Singapore(新加坡国立大学)
Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
Meituan Inc.(美团公司);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
School of Computer Science and Technology, Beijing Institute of Technology, China(北京理工大学计算机科学与技术学院);School of Computer Science and Technology, Huazhong University of Science and Technology, China(华中科技大学计算机科学与技术学院);Independent, China(独立)
Stop Tracking Me! Proactive Defense Against Attribute Inference Attack in LLMs
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs
School of Software, Tsinghua University(清华大学软件学院);School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);Beihang University(北航);Department of Automation, Tsinghua University(清华大学自动化系)
Do LLMs Have the Generalization Ability in Conducting Causal Inference?
College of Intelligence and Computing, Tianjin University(天津大学智能与计算学部);Institute of Applied Psychology, Tianjin University(天津大学应用心理研究所);China Mobile Communication Group Tianjin Co.,Ltd(中国移动通信集团天津有限公司)