We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs. We propose an alternative way of structuring LLMs with weight sharing between layers in subsequent Transformer blocks, along with additional low-rank difference matrices between them. For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules with approximately 30M-40M tokens is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch. We release the resultant models, DeltaLLAMA and DeltaPHI, with a 12% parameter reduction, retaining 90% of the performance of the base Llama and Phi models on common knowledge and reasoning benchmarks. Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed. For example, DeltaPhi 2.9B with a 24% reduction achieves similar average zero-shot accuracies as recovery fine-tuned SlicedPhi 3.3B with a 12% reduction, despite being approximately 400M parameters smaller with no fine-tuning applied. This work provides new insights into LLM architecture design and compression methods when storage space is critical.
DeltaLLM: Delta-Key Sparse Attention for Edge LLMs
Leiden Institute of Advanced Computer Science (LIACS), Leiden University(莱顿先进计算机科学研究所(LIACS)、莱顿大学);Department of Microelectronics, Delft University of Technology(电子系、代尔夫特理工大学)
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));TeleAI of China Telecom(中国电信TeleAI);The Hong Kong Polytechnic University(香港理工大学);The University of Hong Kong(香港大学)
Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform
Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Harbin Institute of Technology(哈尔滨工业大学)
DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models
School of Information Science and Technology, Fudan University(复旦大学信息科学与技术学院);The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室);Baidu USA(美国百度公司);StepFun(阶跃星辰)