We propose a Quantization Guided Training (QGT) method to guide DNN training towards optimized low-bit-precision targets and reach extreme compression levels below 8-bit precision. Unlike standard quantization-aware training (QAT) approaches, QGT uses customized regularization to encourage weight values towards a distribution that maximizes accuracy while reducing quantization errors. One of the main benefits of this approach is the ability to identify compression bottlenecks. We validate QGT using state-of-the-art model architectures on vision datasets. We also demonstrate the effectiveness of QGT with an 81KB tiny model for person detection down to 2-bit precision (representing 17.7x size reduction), while maintaining an accuracy drop of only 3% compared to a floating-point baseline.
This paper studies the merits of applying log-gradient input images to convolutional neural networks (CNNs) for tinyML computer vision (CV). We show that log gradients enable: (i) aggressive 1.5-bit quantization of first-layer inputs, (ii) potential CNN resource reductions, and (iii) inherent robustness to illumination changes (1.7% accuracy loss across 1/32...8 brightness variation vs. up to 10% for JPEG). We establish these results using the PASCAL RAW image data set and through a combination of experiments using neural architecture search and a fixed three-layer network. The latter reveal t…
Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devices
School of Engineering Mathematics and Technology, University of Bristol（布里斯托大学工程数学与技术学院）；School of Natural and Computing Sciences, University of Aberdeen（阿伯丁大学自然与计算科学学院）
Qualcomm AI Research（高通人工智能研究院）；Qualcomm Canada ULC（高通加拿大无限责任公司）；University of Toronto（多伦多大学）
Tiny machine learning (tinyML) has emerged during the past few years aiming to deploy machine learning models to embedded AI processors with highly constrained memory and computation capacity. Low precision quantization is an important model compression technique that can greatly reduce both memory consumption and computation cost of model inference. In this study, we focus on post-training quantization (PTQ) algorithms that quantize a model to low-bit (less than 8-bit) precision with only a small set of calibration data and benchmark them on different tinyML use cases. To achieve a fair compa…
GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
Department of XXX, University of YYY, Location, Country；School of ZZZ, Institute of WWW, Location, Country；Department of Computer Science；Engineering, Seoul National University；Neural Processing Research Center；Samsung AI Lab, Montreal；Google
Multidisciplinary Faculty of Nador, Mohammed Premier University, Oujda 60000, Morocco（摩洛哥奥伊达大学多学科学院）；Laboratory LaSTI, ENSAK, Sultan Moulay Slimane University, Khouribga 54000, Morocco（摩洛哥库尔希布加大学LaSTI实验室）
HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs
School of Software and Microelectronics, Peking University, Beijing, China（软件与微电子学院，北京大学，北京，中国）；School of Computer Science, Peking University, Beijing, China（计算机科学学院，北京大学，北京，中国）
QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；The Chinese University of Hong Kong（香港中文大学）；Huawei Cloud Computing Technologies Co., Ltd（华为云计算技术有限公司）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北京航空航天大学）
Sensitivity-Aware Post-Training Quantization for Deep Neural Networks
South China University of Technology（华南理工大学）；Pazhou Laboratory（Pazhou实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人重点实验室，教育部）
Selective Focus: Investigating Semantics Sensitivity in Post-training Quantization for Lane Detection
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；SenseTime Research（商汤科技研究院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；State Key Laboratory of Complex & Critical Software Environment, Beihang University（北京航空航天大学复杂与关键软件环境国家重点实验室）
Model quantization is a promising approach to compress deep neural networks and accelerate inference, making it possible to be deployed on mobile and edge devices. To retain the high performance of full-precision models, most existing quantization methods focus on fine-tuning quantized model by assuming training datasets are accessible. However, this assumption sometimes is not satisfied in real situations due to data privacy and security issues, thereby making these quantization methods not applicable. To achieve zero-short model quantization without accessing training data, a tiny number of …
Université de Montréal（蒙特利尔大学）；MILA；IMT Atlantique（IMT大西洋高等矿业学院）；Lab-STICC（STICC实验室）；ISCAS 2019（ISCAS 2019（会议名称，不提取））
Convolutional Neural Networks (CNNs) are state-of-the-art in numerous computer vision tasks such as object classification and detection. However, the large amount of parameters they contain leads to a high computational complexity and strongly limits their usability in budget-constrained devices such as embedded devices. In this paper, we propose a combination of a new pruning technique and a quantization scheme that effectively reduce the complexity and memory usage of convolutional layers of CNNs, and replace the complex convolutional operation by a low-cost multiplexer. We perform experimen…
DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Quantization Meets OOD: Generalizable Quantization-aware Training from a Flatness Perspective
SIGS, Tsinghua University（清华大学信息科学技术系）；Key Laboratory of Pervasive Computing, Ministry of Education（教育部 pervasive 计算重点实验室）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；MMLab, The Chinese University of Hong Kong（香港中文大学 MMLab）
Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；China Telecom（中国电信）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Zhongguancun Academy（中关村学院）
Microsoft Research（微软研究院）；Microsoft（微软）；Shanghai Jiao Tong University（上海交通大学）
The combination of Neural Architecture Search (NAS) and quantization has proven successful in automatically designing low-FLOPs INT8 quantized neural networks (QNN). However, directly applying NAS to design accurate QNN models that achieve low latency on real-world devices leads to inferior performance. In this work, we find that the poor INT8 latency is due to the quantization-unfriendly issue: the operator and configuration (e.g., channel width) choices in prior art search spaces lead to diverse quantization efficiency and can slow down the INT8 inference speed. To address this challenge, we…
Tail-Aware Post-Training Quantization for 3D Geometry Models
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China（清华大学深圳国际研究生院，清华大学，深圳，中国）；MMLab, The Chinese University of Hong Kong, Hong Kong, China（MMLab，香港中文大学，香港，中国）；Harbin Institute of Technology Shenzhen, China（哈尔滨工业大学深圳，中国）；Huawei Technology, Shenzhen, China（华为技术，深圳，中国）
Improving Quantization-aware Training of Low-Precision Network via Block Replacement on Full-Precision Counterpart
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；ZJU-UIUC Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究院）
PTQ4ARVG: Post-Training Quantization for AutoRegressive Visual Generation Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Dataset Color Quantization: A Training-Oriented Framework for Dataset-Level Compression
CFAR, Agency for Science, Technology and Research, Singapore（科技研究局CFAR, 新加坡）；IHPC, Agency for Science, Technology and Research, Singapore（科技研究局IHPC, 新加坡）；National University of Singapore（新加坡国立大学）；University of Electronic Science and Technology of China (UESTC)（电子科技大学）；Nanyang Technological University (NTU), Singapore（南洋理工大学, 新加坡）
Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks
The State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；School of Cyber Science and Engineering, Wuhan University（武汉大学网络安全学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；X Digital Dynamics（X数字动力公司）；Tencent Data Platform（腾讯数据平台）
The JPEG image compression algorithm is the most popular method of image compression because of its ability for large compression ratios. However, to achieve such high compression, information is lost. For aggressive quantization settings, this leads to a noticeable reduction in image quality. Artifact correction has been studied in the context of deep neural networks for some time, but the current state-of-the-art methods require a different model to be trained for each quality setting, greatly limiting their practical application. We solve this problem by creating a novel architecture which …
Neural networks are essential components of learning-based software systems. However, their high compute, memory, and power requirements make using them in low resources domains challenging. For this reason, neural networks are often quantized before deployment. Existing quantization techniques tend to degrade the network accuracy. We propose Counter-Example Guided Neural Network Quantization Refinement (CEG4N). This technique combines search-based quantization and equivalence verification: the former minimizes the computational requirements, while the latter guarantees that the network's outp…
CALM: A CKA-Guided Adaptive Layer-Wise Modularization Framework for LLM Quantization
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Zhejiang University（浙江大学）；University of Science and Technology of China（中国科学技术大学）
