Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.
A Survey on Model Compression for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
Centre for Responsible AI(负责任人工智能中心);Indian Institute of Technology Madras(印度理工学院马德拉斯分校);Department of Electrical Engineering(电气工程系);Wadhwani School of Data Science and Artificial Intelligence(Wadhwani数据科学与人工智能学院)
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));CSE, The Hong Kong University of Science and Technology(香港科学与技术大学计算机科学与工程系);Guangzhou HKUST Fok Ying Tung Research Institute(广州HKUST福 Ying Tung研究 institute);Terminus Technologies
EvolKV: Evolutionary KV Cache Compression for LLM Inference
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CAS(复杂系统认知与决策智能重点实验室);ETH Zurich(苏黎世联邦理工学院)
Tensor Methods for Language Models: From Token Representation to Training, Adaptation, Inference, Compression, and Interpretability
Innopolis University(因诺波利斯大学);Federal University of Ceará(塞阿拉联邦大学);Systems Research Institute of Polish Academy of Science(波兰科学院系统研究所);Warsaw University of Technology(华沙理工大学)
NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium
University of California, Merced USA;University of Wisconsin, Madison USA;Yotta Labs \& University of California, Merced USA;University of California, Merced;University of Wisconsin, Madison;Yotta Labs \& University of California, Merced
HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
School of Software Technology, Zhejiang University(浙江大学软件学院);China Mobile (Zhejiang) Research & Innovation Institute(中国移动(浙江)研究院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD与CG国家重点实验室);Geely Automobile Research Institute (Ningbo) Co., Ltd.(吉利汽车研究院(宁波)有限公司);FABU Inc.(FABU公司)
SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
AGI Institute, School of Computer Science, Shanghai Jiao Tong University(AGI研究院,计算机科学学院,上海交通大学);Shanghai Key Laboratory for Intelligent Information Processing, Fudan University(上海智能信息处理重点实验室,复旦大学);State Key Laboratory for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学);School of Artificial Intelligence, Wuhan University(人工智能学院,武汉大学);Institute of Computer Science, University of Göttingen(计算机科学研究所,哥廷根大学)
Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Survey
Renmin University of China(中国人民大学);Artificial Intelligence Innovation and Incubation Institute, Fudan University(复旦大学人工智能创新与孵化院);Shanghai Academy of AI for Science(上海人工智能科学研究院);Department of houmo.ai(houmo.ai部门)
On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
The Hong Kong Polytechnic University, Hong Kong(香港理工大学);Nanyang Technological University, Singapore(南洋理工大学);Chongqing University, Chongqing, China(重庆大学);Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU(未来智能系统中的隐私与安全技术研究中心)