In this paper, we present a simple optimization-based preprocessing technique called Weight Magnitude Reduction (MagR) to improve the performance of post-training quantization. For each linear layer, we adjust the pre-trained floating-point weights by solving an $\ell_\infty$-regularized optimization problem. This process greatly diminishes the maximum magnitude of the weights and smooths out outliers, while preserving the layer's output. The preprocessed weights are centered more towards zero, which facilitates the subsequent quantization process. To implement MagR, we address the $\ell_\infty$-regularization by employing an efficient proximal gradient descent algorithm. Unlike existing preprocessing methods that involve linear transformations and subsequent post-processing steps, which can introduce significant overhead at inference time, MagR functions as a non-linear transformation, eliminating the need for any additional post-processing. This ensures that MagR introduces no overhead whatsoever during inference. Our experiments demonstrate that MagR achieves state-of-the-art performance on the Llama family of models. For example, we achieve a Wikitext2 perplexity of 5.95 on the LLaMA2-70B model for per-channel INT2 weight quantization without incurring any inference overhead.
LoPRo: Enhancing Low-Rank Quantization via Permuted Block-Wise Rotation
Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Aeronautics and Astronautics(北京航空航天大学)
Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study
University of Toronto(多伦多大学);The Hong Kong University of Science and Technology(香港科学与技术大学);Boston College(波士顿学院);Lehigh University(莱特大学);Southern University of Science and Technology(南方科技大学)
Polar Code Based Federated Learning: Convergence Analysis and Resource Allocation
School of Information Science and Engineering, Southeast University(东南大学信息科学与工程学院);National Mobile Communications Research Laboratory, Southeast University(东南大学国家移动通信重点实验室)
Mitigating Adversarial Perturbations for Deep Reinforcement Learning via Vector Quantization
School of Electrical Engineering, Korea Advanced Institute of Science and Technology(韩国科学技术院电气工程学院);Department of Artificial Intelligence, Korea University(高丽大学人工智能系)
Enhancing Convergence, Privacy and Fairness for Wireless Personalized Federated Learning: Quantization-Assisted Min-Max Fair Scheduling
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications(信息与通信工程学院,北京邮电大学);School of Computing, Macquarie University(计算学院,麦考瑞大学);Department of Broadband Communication, Peng Cheng Laboratory(宽带通信部,鹏城实验室);China Telecom, Sichuan Branch(中国电信四川分公司)
Enhancing Quantization-Aware Training on Edge Devices via Relative Entropy Coreset Selection and Cascaded Layer Correction
Hubei Key Laboratory of Trasportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology(湖北交通物联网重点实验室,计算机科学与人工智能学院,武汉理工大学);School of Computer Science, Wuhan University(计算机科学学院,武汉大学)
FedShift: Robust Federated Learning Aggregation Scheme in Resource Constrained Environment via Weight Shifting
Department of Electrical Engineering and Computer Science, University of Stavanger(电气工程与计算机科学系,斯塔万格大学);Department of Computer Convergence Software, Korea University(计算机融合软件系,韩国大学)
AQUILA: Communication Efficient Federated Learning with Adaptive Quantization in Device Selection Strategy
Tsinghua University(清华大学);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Tsinghua-Berkeley Shenzhen Institute(清华-伯克利深圳学院);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Shanghai AI Lab(上海人工智能实验室);George Washington University(乔治华盛顿大学);Ryerson University(瑞尔森大学)
Apollo-Forecast: Overcoming Aliasing and Inference Speed Challenges in Language Models for Time Series Forecasting
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University(同济大学上海智能自主系统研究院);College of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);Ant Group(蚂蚁集团);Huazhong University of Science and Technology(华中科技大学);Macau University of Science and Technology(澳门科技大学)
我们提出了1.58位FLUX,这是第一个成功的方法来量化最先进的文本到图像生成模型FLUX.1-dev,使用1.58位权重(即,{-1,0,+1}中的值),同时保持生成1024 x 1024图像的相当性能。值得注意的是,我们的量化方法在不访问图像数据的情况下运行,仅依赖于FLUX.1-dev模型的自我监督。此外,我们开发了一个针对1.58位操作优化的自定义内核,实现了模型存储减少7.7倍,推理内存减少5.1倍,并改善了推理延迟。GenEval和T2 I Compbench基准测试的广泛评估表明,1.58位FLUX在保持生成质量的同时显着提高计算效率方面是有效的。
Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025
Federal University of Goiás(戈亚斯联邦大学);Federal University of Rio Grande do Norte(北里奥格兰德联邦大学);Aeronautics Institute of Technology(航空技术研究所);Federal University of Mato Grosso(马托格罗索联邦大学)
本文面向 INTERSPEECH 2025 自然条件语音情感识别挑战赛,提出结合先进音频模型、韵律与频谱增强文本特征、F0 量化和集成学习的鲁棒系统,并验证图注意力网络融合有效性,测试集 Macro F1 达 39.79%。