Large language models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their massive parameter counts. While existing compression techniques like pruning can reduce model size, it leads to significant accuracy degradation under high compression ratios. We present a novel perspective inspired by constant folding in compiler optimization. Our approach enables parameter reduction by treating activation functions in LLMs as linear functions.
However, recent LLMs use complex non-linear activations like GELU that prevent direct application of this technique. We propose TARDIS, which enables optimization of LLMs with non-linear activations by partially approximating them with linear functions in frequently occurring input ranges. For outlier inputs, TARDIS employs an online predictor to dynamically fall back to original computations.
Our experiments demonstrate that TARDIS achieves 80% parameter reduction in feed-forward networks, while significantly outperforming state-of-the-art pruning methods Wanda and RIA with up to 65% higher accuracy. In practical deployments for a 7B model, TARDIS achieves 1.6x end-to-end inference speedup when integrated with the vLLM serving system, and 1.4x speedup with the widely adopted HuggingFace implementation, while incurring only a 10.9% accuracy trade-off.
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
School of Mathematical Sciences, Peking University(北京大学数学科学学院);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心);Shanghai Jiao Tong University(上海交通大学);AI for Science Institute, Beijing, China(北京人工智能科学研究院)
FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimation
The Ohio State University(俄亥俄州立大学);Department of Electrical and Computer Engineering(电气与计算机工程系);Department of Industrial Engineering(工业工程系);University of Kentucky(肯塔基大学);Air Force Research Laboratory(空军研究实验室)