Deep neural networks (DNNs) have been proven to outperform classical methods on several machine learning benchmarks. However, they have high computational complexity and require powerful processing units. Especially when deployed on embedded systems, model size and inference time must be significantly reduced. We propose SYMOG (symmetric mixture of Gaussian modes), which significantly decreases the complexity of DNNs through low-bit fixed-point quantization. SYMOG is a novel soft quantization method such that the learning task and the quantization are solved simultaneously. During training the weight distribution changes from an unimodal Gaussian distribution to a symmetric mixture of Gaussians, where each mean value belongs to a particular fixed-point mode. We evaluate our approach with different architectures (LeNet5, VGG7, VGG11, DenseNet) on common benchmark data sets (MNIST, CIFAR-10, CIFAR-100) and we compare with state-of-the-art quantization approaches. We achieve excellent results and outperform 2-bit state-of-the-art performance with an error rate of only 5.71% on CIFAR-10 and 27.65% on CIFAR-100.
MoQE: Improve Quantization Model performance via Mixture of Quantization Experts
Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；North University of China（北方大学）
University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）
Deep convolutional neural networks (CNNs) have proven highly effective for visual recognition, where learning a universal representation from activations of convolutional layer plays a fundamental problem. In this paper, we present Fisher Vector encoding with Variational Auto-Encoder (FV-VAE), a novel deep architecture that quantizes the local activations of convolutional layer in a deep generative model, by training them in an end-to-end manner. To incorporate FV encoding strategy into deep generative models, we introduce Variational Auto-Encoder model, which steers a variational inference an…
MEC-Quant: Maximum Entropy Coding for Extremely Low Bit Quantization-Aware Training
Faculty of Information Technology, Beijing University of Technology（信息技术学院，北京理工大学）；University of Chinese Academy of Sciences, Chinese Academy of Sciences (CAS)（中国科学院大学，中国科学院）；Institute of Computing Technology, CAS（计算技术研究所，中国科学院）
Timestep-Aware SVDQuant-GPTQ for W4A4 Quantization of Wan2.2-I2V
National Engineering Research Center for Big Data Technology and System（大数据技术与系统国家工程研究中心）；Services Computing Technology and System Lab（服务计算技术与系统实验室）；Cluster and Grid Computing Lab（集群与网格计算实验室）；School of Computer Science and Technology（计算机科学与技术学院）；Huazhong University of Science and Technology（华中科技大学）
CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
Courant Institute of Mathematical Sciences, New York University（纽约大学库朗数学科学研究所）；Tandon School of Engineering, New York University（纽约大学坦登工程学院）；Cerebras Systems Inc.（Cerebras系统公司）
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
University of Arizona（亚利桑那大学）；Duke University（杜克大学）；Sungkyunkwan University（成均馆大学）；Panasonic AI Lab（松下人工智能实验室）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Xiamen University（厦门大学多媒体可信感知与高效计算重点实验室）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（伯克利国家实验室）；University of California, Berkeley（加州大学伯克利分校）；Liquid AI
MMQ: Multimodal Mixture-of-Quantization Tokenization for Semantic ID Generation and User Behavioral Adaptation
Alibaba Group（阿里巴巴集团）；Peking University（北京大学）；Beijing University of Aeronautics and Astronautics（北京航空航天大学）；Wuhan University, School of Computer Science（武汉大学计算机学院）
MC#: Mixture Compressor for Mixture-of-Experts Large Models
Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电子与电气工程系）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；NVIDIA Research（NVIDIA研究）
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Nanjing University of Science and Technology（南京理工大学）；AIRIA
Robert Bosch GmbH（罗伯特·博世有限公司）；University of Freiburg（弗莱堡大学）
Due to their high computational complexity, deep neural networks are still limited to powerful processing units. To promote a reduced model complexity by dint of low-bit fixed-point quantization, we propose a gradient-based optimization strategy to generate a symmetric mixture of Gaussian modes (SGM) where each mode belongs to a particular quantization stage. We achieve 2-bit state-of-the-art performance and illustrate the model's ability for self-dependent weight adaptation during training.
MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
Department of Computer Science and Software Engineering, Auburn University, Auburn, AL, USA（计算机科学与软件工程系，阿伯拉罕大学，阿伯丁，阿拉巴马州，美国）；Department of Information Management, National Central University, Taoyuan, Taiwan（信息管理系，国立中央大学，桃园，台湾）
Uncertainty Makes It Stable: Curiosity-Driven Quantized Mixture-of-Experts
MIT Critical Data（MIT关键数据）；Université de Rennes（里昂大学）；Technetium Engineering（Technetium工程）；Institución Universitaria Colegio Mayor del Cauca（科瓦卡大学学院）；CeADAR - Ireland’s Centre for AI（爱尔兰人工智能中心）；University College Dublin（都柏林大学）；Tsinghua University（清华大学）；Beth Israel Deaconess Medical Center（贝瑟尔以色列德acons医院）；Harvard T.H. Chan School of Public Health（哈佛T.H.陈公共卫生学院）
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）
An efficient, and intuitive algorithm is presented for the identification of speakers from a long dataset (like YouTube long discussion, Cocktail party recorded audio or video).The goal of automatic speaker identification is to identify the number of different speakers and prepare a model for that speaker by extraction, characterization and speaker-specific information contained in the speech signal. It has many diverse application specially in the field of Surveillance, Immigrations at Airport, cyber security, transcription in multi-source of similar sound source, where it is difficult to ass…
