Layer-wise LoRA fine-tuning: a similarity metric approach
Keith Ando Ogawa, Bruno Lopes Yamamoto, Lucas Lauton de Alcantara, Lucas Pellicer, Rosimeire Pereira Costa, Edson Bollis, Anna Helena Reali Costa, Artur Jordao
备注：
Accepted at the International Conference on Pattern Recognition (ICPR 2026)
Pre-training Large Language Models (LLMs) on web-scale datasets becomes fundamental for advancing general-purpose AI. In contrast, enhancing their predictive performance on downstream tasks typically involves adapting their knowledge through fine-tuning. Parameter-efficient fine-tuning techniques, such as Low-Rank Adaptation (LoRA), aim to reduce the computational cost of this process by freezing the pre-trained model and updating a smaller number of parameters. In comparison to full fine-tuning, these methods achieve over 99\% reduction in trainable parameter count, depending on the configuration. Unfortunately, such a reduction may prove insufficient as LLMs continue to grow in scale. In this work, we address the previous problem by systematically selecting only a few layers to fine-tune using LoRA or its variants. We argue that not all layers contribute equally to the model adaptation. Leveraging this, we identify the most relevant layers to fine-tune by measuring their contribution to changes in internal representations. Our method is orthogonal to and readily compatible with existing low-rank adaptation techniques. We reduce the trainable parameters in LoRA-based techniques by up to 50\%, while maintaining the predictive performance across different models and tasks. Specifically, on encoder-only architectures, this reduction in trainable parameters leads to a negligible predictive performance drop on the GLUE benchmark. On decoder-only architectures, we achieve a small drop or even improvements in the predictive performance on mathematical problem-solving capabilities and coding tasks. Finally, this effectiveness extends to multimodal models, for which we also observe competitive results relative to fine-tuning with LoRA modules in all layers. Code is available at: https://github.com/c2d-usp/Layer-wise-LoRA-with-CKA
Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation
Huazhong University of Science and Technology（华中科技大学）；Shenzhen Technology University（深圳技术大学）；City University of Hong Kong（香港城市大学）；Shenzhen University（深圳大学）
School of Computing University of Connecticut（计算机学院 美国康涅狄格大学）；NEC Labs America（NEC美国实验室）；Dept. of Computer Science Georgia State University（计算机科学系 美国佐治亚州立大学）
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
Department of Mathematics, University of California San Diego（数学系，加州大学圣地亚哥分校）；Department of Mathematics and Halıcıoğlu Data Science Institute, University of California San Diego（数学系和Halıcıoğlu数据科学研究所，加州大学圣地亚哥分校）
Dual-LoRA and Quality-Enhanced Pseudo Replay for Multimodal Continual Food Learning
College of Computer Science and Artificial Intelligence, Fudan University, China（复旦大学计算机科学与人工智能学院）；Singapore Management University, Singapore（新加坡国立管理学院）
Two Stage Wireless Federated LoRA Fine-Tuning with Sparsified Orthogonal Updates
Department of Electrical and Computer Engineering, and the Institute of New Media and Communications, Seoul National University (SNU)（电气与计算机工程系，以及新媒体与通讯研究所，首尔国立大学）
X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
Academy for Science and Design (ASD)（科学设计学院）；Massachusetts Institute of Technology (MIT)（麻省理工学院）
Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMs
School of Electronics and Computer Science, University of Southampton, UK（索姆塞特大学电子与计算机科学学院）；Department of Computer Science, University of Liverpool, UK（利弗pool大学计算机科学系）
LoRA-PAR: A Flexible Dual-System LoRA Partitioning Approach to Efficient LLM Fine-Tuning
School of Politics and Public Administration, South China Normal University（华南师范大学政治与公共管理学院）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang institute of computing technology, Chinese academy of sciences（中国科学院沈阳计算技术研究所）
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China（计算机科学与技术学院，中国科学技术大学）；Suzhou Institute for Advanced Research, University of Science and Technology of China, China（苏州先进研究院，中国科学技术大学）
Behind the Scenes: Mechanistic Interpretability of LoRA-adapted Whisper for Speech Emotion Recognition
Shanghai Institute of Artificial Intelligence for Education, East China Normal University, China（上海人工智能教育研究院，华东师范大学，中国）；School of Computer Science and Technology, East China Normal University, China（计算机科学与技术学院，华东师范大学，中国）；Boin Hearing Technology (Shanghai) Co., LTD, China（博英听力技术（上海）有限公司，中国）；Department of Computing Science, University of Aberdeen, UK（计算科学系，阿伯丁大学，英国）
