Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models.
DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Institute of Automation, CAS（中国科学院自动化研究所）；City University of Hong Kong（香港城市大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；Graduate School of Information Science, Hokkaido University（北海道大学信息科学研究院）
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；IST Austria（奥地利科学技术研究所）；Neural Magic, Inc.（Neural Magic公司）
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
School of Computer and Communication Sciences, EPFL, Switzerland（瑞士联邦理工学院计算机与通信科学学院）；Huawei, Switzerland（华为公司）；Swiss Data Science Center, ETHZ & EPFL, Switzerland（瑞士数据科学中心，苏黎世联邦理工学院与联邦理工学院）
Finedeep: Mitigating Sparse Activation in Dense LLMs via Multi-Layer Fine-Grained Experts
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Chinese Academy of Sciences（中国科学院）；Kuaishou Technology（快手科技）；Tsinghua University（清华大学）
LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering
York University（约克大学）；Vector Institute（向量研究所）；Connected Minds（连接思维公司）；Emory University（埃默里大学）；Wilfrid Laurier University（威尔弗里德·劳里埃大学）；University of Toronto（多伦多大学）；University of Guelph（圭尔夫大学）；Monark Health（莫纳克健康公司）；CIFAR Solution Network（加拿大高级研究院解决方案网络）
Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Models
Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能和数据分析实验室）；King Abdullah University of Science and Technology (KAUST)（卡布斯科学技术大学）；University of Copenhagen（哥本哈根大学）；University of Macau（澳门大学）
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学院，奥布达大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Wuhan AI Research（武汉人工智能研究院）；Objecteye Inc.（智视科技）
Mitigating Overthinking in Large Reasoning Models via Manifold Steering
Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；College of AI, Tsinghua University（清华大学人工智能学院）；Shanghai Qi Zhi Institute（上海启智研究所）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺依曼信息学院，奥布达大学）
