We show that model compression can improve the population risk of a pre-trained model, by studying the tradeoff between the decrease in the generalization error and the increase in the empirical risk with model compression. We first prove that model compression reduces an information-theoretic bound on the generalization error; this allows for an interpretation of model compression as a regularization technique to avoid overfitting. We then characterize the increase in empirical risk with model compression using rate distortion theory. These results imply that the population risk could be improved by model compression if the decrease in generalization error exceeds the increase in empirical risk. We show through a linear regression example that such a decrease in population risk due to model compression is indeed possible. Our theoretical results further suggest that the Hessian-weighted $K$-means clustering compression approach can be improved by regularizing the distance between the clustering centers. We provide experiments with neural networks to support our theoretical assertions.
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
Nanjing University(南京大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学);Hong Kong University of Science and Technology(香港科学大学)
Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
The Hong Kong Polytechnic University(香港理工大学);The University of British Columbia(不列颠哥伦比亚大学);Hohai University(河海大学);Huazhong University of Science and Technology(华中科技大学);University of Waterloo(滑铁卢大学)
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);King Abdullah University of Science and Technology(阿卜杜拉国王科技大学);Dell Technologies(戴尔科技集团);Comenius University(夸美纽斯大学)
CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimization
King Abdullah University of Science and Technology (KAUST)(阿卜杜拉国王科技大学);SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI)(SDAIA-KAUST数据科学与人工智能卓越中心);Brno University of Technology(布尔诺理工大学);Kempelen Institute of Intelligent Technologies (KInIT)(肯佩伦智能技术研究所)
Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects
Department of Mathematics and Statistics, McGill University, Montreal, Canada(数学与统计学系,麦吉尔大学,蒙特利尔,加拿大);Department of Mathematics(数学系);Statistics, McGill University, Montreal, Canada(统计学,麦吉尔大学,蒙特利尔,加拿大);Google Deepmind, Mountain View, United States of America(DeepMind,谷歌,山景城,美国)
Compression-Based Regularization with an Application to Multi-Task Learning
Facultad de Ingeniería - Universidad de Buenos Aires(布宜诺斯艾利斯大学工程学院);CSC-CONICET(阿根廷国家科学与技术研究理事会计算科学中心);CentraleSupélec(巴黎中央理工-高等电力学院);CNRS(法国国家科学研究中心);Université Paris-Sud(巴黎南大学)