Large Language Models (LLMs) stand out for their impressive performance in intricate language modeling tasks. However, their demanding computational and memory needs pose obstacles for broad use on edge devices. Quantization is then introduced to boost LLMs' on-device efficiency. Recent works show that 8-bit or lower weight quantization is feasible with minimal impact on end-to-end task performance, while the activation is still not quantized. On the other hand, mainstream commodity edge devices still struggle to execute these sub-8-bit quantized networks effectively. In this paper, we propose Agile-Quant, an activation-guided quantization framework for popular Large Language Models (LLMs), and implement an end-to-end accelerator on multiple edge devices for faster inference. Considering the hardware profiling and activation analysis, we first introduce a basic activation quantization strategy to balance the trade-off of task performance and real inference speed. Then we leverage the activation-aware token pruning technique to reduce the outliers and the adverse impact on attentivity. Ultimately, we utilize the SIMD-based 4-bit multiplier and our efficient TRIP matrix multiplication to implement the accelerator for LLMs on the edge. We apply our framework on different scales of LLMs including LLaMA, OPT, and BLOOM with 4-bit or 8-bit for the activation and 4-bit for the weight quantization. Experiments show that Agile-Quant achieves simultaneous quantization of model weights and activations while maintaining task performance comparable to existing weight-only quantization methods. Moreover, in the 8- and 4-bit scenario, Agile-Quant achieves an on-device speedup of up to 2.55x compared to its FP16 counterparts across multiple edge devices, marking a pioneering advancement in this domain. Code: https://github.com/shawnricecake/agile-quant
AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models
School of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University（吉林大学符号计算与知识工程教育部重点实验室）
STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization
Qualcomm AI Research（高通人工智能研究）
量化是减少生成式人工智能模型的推理延迟、功耗和内存占用的关键方法。然而，当激活被量化为低于8位时，准确度通常急剧下降。最近的工作表明，可逆线性变换（例如旋转）可以通过重新参数化特征通道和权重来帮助量化。在本文中，我们提出了\textit{Sequence Transformation and Mixed Precision}（STaMP）的量化，一种新的策略，适用于线性变换沿\textit{sequence}维利用语言和视觉数据的强局部相关性。通过以更高的精度在每个中间激活中保持少量的令牌，我们可以在较低的（平均）激活位宽下保持模型的准确性。我们评估STaMP最近的LVM和LLM架构，证明它显着提高了低位宽激活量化和补充建立激活和权重量化方法，包括最近的功能转换。
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学院，奥布达大学）
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Central Media Technology Institute, Huawei Technologies Ltd.（华为技术有限公司媒体技术研究所）
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
School of Software Technology, Zhejiang University（浙江大学软件学院）；Ant Group（蚂蚁集团）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；China Mobile (Zhejiang) Research & Innovation Institute（中国移动（浙江）研究院）；Alibaba Cloud Computing（阿里云计算）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hebei Province Key Laboratory of Big Data Science and Intelligent Technology（河北省大数据科学与智能技术重点实验室）
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；IST Austria（奥地利科学技术研究所）；Neural Magic, Inc.（Neural Magic公司）
