Modern recurrent layers are emerging as a promising path toward edge deployment of foundation models, especially in the context of large language models (LLMs). Compressing the whole input sequence in a finite-dimensional representation enables recurrent layers to model long-range dependencies while maintaining a constant inference cost for each token and a fixed memory requirement. However, the practical deployment of LLMs in resource-limited environments often requires further model compression, such as quantization and pruning. While these techniques are well-established for attention-based models, their effects on recurrent layers remain underexplored.
In this preliminary work, we focus on post-training quantization for recurrent LLMs and show that Mamba models exhibit the same pattern of outlier channels observed in attention-based LLMs. We show that the reason for the difficulty of quantizing SSMs is caused by activation outliers, similar to those observed in transformer-based LLMs. We report baseline results for post-training quantization of Mamba that do not take into account the activation outliers and suggest first steps for outlier-aware quantization.
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；IST Austria（奥地利科学技术研究所）；Neural Magic, Inc.（Neural Magic公司）
MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models
institutetext: MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models Yue Wu Changyuan Wang Zixuan Wang Shilin Ma Yansong Tang（机构文本：MorphoQuant：多模态大语言模型的模态感知量化 Yue Wu 王昌元 王梓轩 马世林 唐彦松）
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
University of Arizona（亚利桑那大学）；Duke University（杜克大学）；Sungkyunkwan University（成均馆大学）；Panasonic AI Lab（松下人工智能实验室）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hebei Province Key Laboratory of Big Data Science and Intelligent Technology（河北省大数据科学与智能技术重点实验室）
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
VCIP, College of Computer Science, Nankai University（南开大学计算机科学学院VCIP）；D-ITET, ETH Zürich（苏黎世联邦理工学院D-ITET）；OPPO Research Institute（OPPO研究院）；Department of Computing, Hong Kong Polytechnic University（香港理工大学计算机系）
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
ZIP Lab, Monash University（莫纳什大学ZIP实验室）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）；School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）；University of Science and Technology Beijing（北京科技大学）
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
School of Software Technology, Zhejiang University（浙江大学软件学院）；Ant Group（蚂蚁集团）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；China Mobile (Zhejiang) Research & Innovation Institute（中国移动（浙江）研究院）；Alibaba Cloud Computing（阿里云计算）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence（人机混合增强智能国家重点实验室）；Institute of Artificial Intelligence and Robotics（人工智能与机器人研究院）；Xi’an Jiaotong University（西安交通大学）
Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview
Peking University（北京大学）；Nankai University（南开大学）；Beijing Institute of Technology（北京理工大学）；University of Leeds（利兹大学）；Harbin University of Commerce（哈尔滨商业大学）
Pushing the Limits of Block Rotations in Post-Training Quantization
Advanced Micro Devices, Inc. (AMD)（Advanced Micro Devices公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Norwegian University of Science（挪威科学与技术大学）
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学校，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学系，奥布达大学）
Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels
Department of Computer Science, Iowa State University（伊阿华州立大学计算机科学系）；Department of Civil, Construction & Environmental Engineering, Iowa State University（伊阿华州立大学土木、建设与环境工程系）
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；School of Engineering Math and Technology, University of Bristol（布里斯托大学工程数学与技术学院）
Benford's Law as a Distributional Prior for Post-Training Quantization of Large Language Models
Postgraduate Program in Computer Science, Federal University of Ouro Preto（计算机科学硕士课程，联邦大学奥鲁普里托）；Computing Department, Federal University of Ouro Preto（计算部门，联邦大学奥鲁普里托）
CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Cerebras Systems Inc.（Cerebras系统公司）
