Model training requires significantly more memory, compared with inference. Parameter efficient fine-tuning (PEFT) methods provide a means of adapting large models to downstream tasks using less memory. However, existing methods such as adapters, prompt tuning or low-rank adaptation (LoRA) either introduce latency overhead at inference time or achieve subpar downstream performance compared with full fine-tuning. In this work we propose Random Subspace Adaptation (ROSA), a method that outperforms previous PEFT methods by a significant margin, while maintaining a zero latency overhead during inference time. In contrast to previous methods, ROSA is able to adapt subspaces of arbitrarily large dimension, better approximating full-finetuning. We demonstrate both theoretically and experimentally that this makes ROSA strictly more expressive than LoRA, without consuming additional memory during runtime. As PEFT methods are especially useful in the natural language processing domain, where models operate on scales that make full fine-tuning very expensive, we evaluate ROSA in two common NLP scenarios: natural language generation (NLG) and natural language understanding (NLU) with GPT-2 and RoBERTa, respectively. We show that on almost every GLUE task ROSA outperforms LoRA by a significant margin, while also outperforming LoRA on NLG tasks. Our code is available at https://github.com/rosa-paper/rosa
Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptation
Shenzhen University（深圳大学）；Hong Kong University of Science（香港科学大学）；Guangdong Laboratory of Artificial Intelligence（广东人工智能与数字经济实验室）；University of Science（科学大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Carleton University（卡尔顿大学）
Unifying Detection and Adaptation in Task-Free Continual Learning
School of Control Science and Engineering, Shandong University（山东大学控制科学与工程学院）；The Hong Kong University of Science and Technology（香港科技大学）；Shandong University（山东大学）；Qilu Hospital of Shandong University（山东大学齐鲁医院）
The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation
School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；Hong Kong University of Science and Technology（香港科技大学）；Frontier Robotics（前沿机器人）
Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
Poznan University of Technology（波兹南理工大学）；IDEAS NCBR；NASK - National Research Institute（NASK国家研究所）；Gdańsk University of Technology（格但斯克理工大学）；Graz University of Technology（格拉茨理工大学）
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China（教育部知识驱动的人机智能工程研究中心）；International Center of Future Science, Jilin University（吉林大学未来科学国际合作中心）
Subspace Identification for Multi-Source Domain Adaptation
Guangdong University of Technology（广东工业大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Carnegie Mellon University（卡内基梅隆大学）；Shantou University（汕头大学）
Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education（交通大数据与人工智能重点实验室（北京交通大学），教育部）；School of Computer Science and Technology, Beijing Jiaotong University（北京交通大学计算机科学与技术学院）
Safe-Subspace Pseudo-Label Refinement for Source-Free Graph Domain Adaptation
Department of Computer Science and Engineering, Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；College of Computer, National University of Defense Technology（国防科技大学计算机学院）；Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；The Education University of Hong Kong（香港教育大学）
Wuhan University（武汉大学）；ByteDance（字节跳动）；National University of Defense Technology（国防科技大学）；Nanyang Technological University（南洋理工大学）；The AGH University of Krakow（克拉科夫AGH大学）
KeepLoRA: Continual Learning with Residual Gradient Adaptation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China（教育部计算机网络与信息集成重点实验室）；School of Software Technology, Zhejiang University（浙江大学软件学院）
KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education（东南大学计算机网络和信息集成教育部重点实验室）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning
Department of Computer Science（计算机科学系）；Information Engineering, National Cheng Kung University（信息工程系，国立成功大学）；Department of Artificial Intelligence, Chang Gung University（人工智能系，长庚大学）；Department of Computer Science, National Tsing Hua University（计算机科学系，国立清华大学）
