High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensionality also introduces considerable model parameters and prohibitively high model storage and memory requirements, which is particularly unaffordable for low-end devices. Targeting no extra training data and insufficient computation cases, we propose a training-free model compression approach based on the Tensor-Train Decomposition (TTD), whereby each pre-trained token embedding is converted into a lower-dimensional Matrix Product State (MPS). We then comprehensively investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e. Raspberry Pi). Taking GPT family models (i.e. GPT-2 and CerebrasGPT) as case studies, our approach theoretically results in $46.89\%$ fewer parameters of the entire model, with a compression ratio $39.38\times$ - $65.64\times$ for the embedding layers. With different hyperparameter choices, the model compressed with our approach can achieve a comparable language task performance to the original model with around $2.0\times$ embedding layer compression. This empirically proves the existence of low-rank structure in GPT family models, and demonstrates that about half of the parameters in the embedding layers are redundant.
SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices
Department of Computer Science, Aalborg University Copenhagen（奥尔堡大学哥本哈根分校计算机科学系）；MaLGa-DIBRIS, University of Genoa（热那亚大学MaLGa-DIBRIS）；INFN, Sezione di Genova（国家核物理研究所热那亚分部）；European Organization for Nuclear Research (CERN)（欧洲核子研究中心）；Ceva, Inc.（Ceva公司）
From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition
DeepLang AI；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Jiaotong University（北京交通大学）
NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium
University of California, Merced USA；University of Wisconsin, Madison USA；Yotta Labs \& University of California, Merced USA；University of California, Merced；University of Wisconsin, Madison；Yotta Labs \& University of California, Merced
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
School of Information Technology and Management, University of International Business and Economics（对外经济贸易大学信息技术与管理学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）；School of Information, Renmin University of China（中国人民大学信息学院）
Break Through the Compression Bottleneck: From Theory to Practice
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Compressing Transformer Language Models via Matrix Product Operator Decomposition: A Case Study on PicoGPT
Institut für Theoretische Physik, Leibniz Universität Hannover（莱布尼茨汉诺威大学理论物理研究所）；VTT Technical Research Centre of Finland（芬兰VTT技术研究中心）；National High Magnetic Field Laboratory（国家高磁场实验室）
