Quantization-Robust LLM Unlearning via Low-Rank Adaptation
Kunumi Institute, Brazil（Kunumi研究所，巴西）
João Vitor Boer Abitante, Joana Meneguzzo Pasquali, Luan Fonseca Garcia, Ewerton de Oliveira, Thomas da Silva Paula, Rodrigo C. Barros, Lucas S. Kupssinskü
Large Language Model (LLM) unlearning aims to remove targeted knowledge from a trained model, but practical deployments often require post-training quantization (PTQ) for efficient inference. However, aggressive low-bit PTQ can mask unlearning updates, causing quantized models to revert to pre-unlearning behavior. We show that standard full-parameter fine-tuning often induces parameter changes that are too small to survive 4-bit quantization. We propose quantization-robust unlearning via low-rank adaptation (LoRA): we freeze the base model and concentrate unlearning into trainable adapters so that the effective update is preserved after quantization. On Llama-2-7B evaluated with MUSE dataset (BOOKS and NEWS), LoRA improves 4-bit utility by up to 7.93 points (NPO+GDR on BOOKS: 50.17 to 58.10) and yields higher 4-bit utility on NEWS for GA+GDR (40.06 to 44.82, increase of 4.76). LoRA also substantially reduces privacy leakage under 4-bit PTQ, e.g., for GA+KLR on BOOKS, PrivLeak moves from -25.68 to -5.86 (closer to ideal 0), while maintaining strong forgetting (VerMem and KnowMem near 0). Thus, using LoRA for Machine Unlearning is beneficial for scenarios where quantization is necessary for model deployment.
NAVER CLOVA；KAIST（韩国科学技术院）；NAVER AI Lab（NAVER人工智能实验室）；SNU AIIS（首尔大学人工智能研究所）
There are growing interests in adapting large-scale language models using parameter-efficient fine-tuning methods. However, accelerating the model itself and achieving better inference efficiency through model compression has not been thoroughly explored yet. Model compression could provide the benefits of reducing memory footprints, enabling low-precision computations, and ultimately achieving cost-effective inference. To combine parameter-efficient adaptation and model compression, we propose AlphaTuning consisting of post-training quantization of the pre-trained language model and fine-tuni…
Storage Is Not Strategy: State-Conditioned Support Control for LLM Unlearning
Southeast University（东南大学）；University of Sydney（悉尼大学）；Northwestern University（西北大学）；Together AI；The Chinese University of Hong Kong Shenzhen（香港中文大学（深圳））
Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearning
School of Software Technology, Zhejiang University（浙江大学软件技术学院）；Zhejiang University（浙江大学）；Hefei University of Technology（合肥工业大学）；Institute of Fundamental and Transdisciplinary Research, Zhejiang University（浙江大学基础与交叉学科研究院）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江区）区块链与数据安全研究院）；The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室）；Zhejiang Key Laboratory of Big Data Intelligent Computing（浙江省大数据智能计算重点实验室）
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Yuanpei College, Peking University, Beijing, China（北京大学元培学院）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））
Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Pittsburgh（匹兹堡大学）；University of Arizona（亚利桑那大学）；University of Central Florida（中佛罗里达大学）
TTAQ: Towards Stable Post-training Quantization in Continuous Domain Adaptation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Inspur Yunzhou Industrial Internet Co., Ltd.（浪潮云洲工业互联网有限公司）
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
Department of Mathematics, University of California San Diego（数学系，加州大学圣地亚哥分校）；Department of Mathematics and Halıcıoğlu Data Science Institute, University of California San Diego（数学系和Halıcıoğlu数据科学研究所，加州大学圣地亚哥分校）
DoTA: Weight-Decomposed Tensor Adaptation for Large Language Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；University of International Business and Economics, China（对外经济贸易大学）；XiaoMi AI Lab, China（小米AI实验室）
Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance
National Key Laboratory of Parallel and Distributed Computing（平行与分布式计算国家重点实验室）；College of Computer Science and Technology（计算机科学与技术学院）；National University of Defense Technology（国防科技大学）
HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization
Department of Engineering, King’s College London（伦敦国王学院工程系）；Advanced Technology Group, Ericsson Inc.（爱立信公司先进科技组）；Institute for Digital Communication, Friedrich-Alexander-Universität Erlangen-Nürnberg（埃尔兰根-纽伦堡弗里德里希-亚历山大大学数字通信研究所）
LoRAQuant: Mixed-Precision Quantization of LoRA to Ultra-Low Bits
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta（计算科学系及阿尔伯塔人工智能研究所（Amii），阿尔伯塔大学）；RBC Borealis；Canada CIFAR AI Chair（加拿大CIFAR人工智能 chair）
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
Beacom College of Computer and Cyber Sciences, Dakota State University（达科他州立大学比科姆计算机与网络科学学院）；Mathematics and Computer Science, Augustana College（奥古斯塔纳学院数学与计算机科学系）
